Matrix determinant contradicts corresponding box volume – how is it possible?

The given vectors do not form a cube, so using the cube's volume formula is invalid. In particular, the vectors would need to be orthogonal (dot product zero for all pairs) for said formula to be applied, but here they are not. Indeed, the angle between each of the pairs here is 60°.

Your 3 vectors are related to a cube, but they do not form the cube edges. They are the diagonals of 3 adjacent faces:

I made a gif that shows the transformation. At the beginning you see the unit cube and at the end of the animation you see the unit cube as transformed by the matrix $$\begin{bmatrix}1 & 0 & 1\\ 1 & 1 & 0\\ 0 & 1 & 1\\\end{bmatrix}$$ As you stated this shape is not a cube but a parallelepiped so you can't calculate the area by multiplying the length of the spanning vectors. So what is the volume of this parallelepiped? There are a couple of ways to calculate this. It turns out the determinant of this matrix is one way, remembering that the determinant can be negative if the shape if flipped. This way is actually kind of tricky to proof for 3D matrices though. In two dimensions the determinant corresponds to the area of a parallelogram which you can proof geometrically.

Finally note that each unit vector gets mapped to their corresponding column vector in the matrix and that all parallel lines remain parallel. This helps you visualise matrix transformations.