Mouse controlled first person movement JS

It's not quite clear to me what your high level goal is. It sounds like you're trying to implement a Counterstrike-like game in JS and CSS. Which is awesome! For the rest of this answer, I'm going to assume that you are trying to do something like that.

Realistically, you must use the Pointer Lock API. Otherwise, you won't be able to turn around by only moving the mouse left. You'll hit the edge of the browser window and stop turning. The browser support isn't great, but it's by far a better experience for the gamer!

In order to render your world with CSS transforms, you need to do a complicated series of transforms to generate the matrix for every side of every object visible in the game world. This is because the browser's perspective is always looking directly along the Z axis. So in order to animate things "around" the viewer's eye, you have to translate and rotate them around. After a bit of poking around, I came to the conclusion that doing all the transforms in CSS is prohibitively slow (and complicated!). But never fear, there's another way! WebGL or Canvas to the rescue!

Take a look at Isaac Sukin's game Nemesis. It's an excellent example, and he's written a tutorial to come up with something similar! The library it's based on, Three.js, is very widely used and has a very understandable API. It takes almost all of the hard part out, and lets you just make a 3D world!

Good luck with the game!

Using quaternions is really easier. I found an implementation in Google closure library so I made an example (also, check the jsFiddle):

goog.require('goog.vec.Quaternion');

var velocity = 0.5;

var lastX = null;
var lastY = null;
var angleX = 0;
var angleY = 0;

$(document).on('mousemove', function (e) {
    if (lastX == null) lastX = e.pageX;
    if (lastY == null) lastY = e.pageY;

    angleX += (e.pageX - lastX) * velocity * Math.PI / 180;
    angleY += (e.pageY - lastY) * velocity * Math.PI / 180;

    lastX = e.pageX;
    lastY = e.pageY;

    var quat = goog.vec.Quaternion.concat(
        goog.vec.Quaternion.fromAngleAxis(angleX, [0, 1, 0], []),
        goog.vec.Quaternion.fromAngleAxis(-angleY, [1, 0, 0], []), []);

    var matrix = goog.vec.Quaternion.toRotationMatrix4(quat, []);

    $("#transformer").css({
        webkitTransform: "matrix3d(" + matrix.join(",") + ")"
    });
});

Pure JavaScript is mostly better than libraries (unless it's a "Code less do more" thing),
since you can understand what your code really does.

This is my entire JavaScript code:

var velocity = 0.5;

document.onmousemove = function(e) {
    var angleX = e.pageY * velocity * Math.PI / 180;
    var angleY = e.pageX * velocity * Math.PI / 180;
    document.getElementById('transformer').style.webkitTransform = 'matrix3d(' + Math.cos(-angleY) + ',0,' + Math.sin(-angleY) + ',0,' + (Math.sin(angleX)*Math.sin(-angleY)) + ',' + Math.cos(angleX) + ',' + (-Math.sin(angleX)*Math.cos(-angleY)) + ',0,' + (-Math.cos(angleX)*Math.sin(-angleY)) + ',' + Math.sin(angleX) + ',' + (Math.cos(angleX)*Math.cos(-angleY)) + ',0,0,0,0,1)';
};

And this is the fiddle.

It works!

(I even made an example of this using the Pointer Lock API: fiddle (click the square to begin)

---

Explanation:

First, a velocity variable to easily set the rotation speed.
Then, a mousemove event which has the two rotation variabls set.
The last line is to convert from rotateX and rotateY transformations, to matrix3d as requested. This Stackoverflow question helped me get to the following solution.

---

rotateX(angleX) is equal to the following matrix:

1               0               0               0


0               cos(angleX)     -sin(angleX)    0


0               sin(angleX)     cos(angleX)     0


0               0               0               1

rotateY(angleY) is equal to the following matrix:

cos(angleY)     0               sin(angleY)     0


0               1               0               0


-sin(angleY)    0               cos(angleY)     0


0               0               0               1

And to use them both together, you need to multiply the two matrices. So I wrote a small JavaScript tool to give me the calculation I need to do to get the result of this multiplication.

The result:

cos(angleY)               sin(angleX)*sin(angleY)   cos(angleX)*sin(angleY)   0



0                         cos(angleX)               -sin(angleX)              0



-sin(angleY)              sin(angleX)*cos(angleY)   cos(angleX)*cos(angleY)   0



0                         0                         0                         1

And that's the way to convert rotateX and rotateY to matrix3d.

Hope it helps :)