Find a matrix and analytic formula for a linear map $f: \mathbb R^{4} \rightarrow \mathbb R^{3}$

$\newcommand{\ran}{\operatorname{ran}}\newcommand{\R}{\mathbb{R}}$I guess you want a matrix which correspond with the linear map with respect to the standard basis.

First of all, let me tell you that you cannot find a unique solution to this exercise. So at some point you have to choose some properties of your solution.

Let $v_1 = \left(\begin{smallmatrix}1 \\ 2\\3 \\ 1\end{smallmatrix}\right)$ and $v_2 = \left(\begin{smallmatrix}1\\5\\4\\1\end{smallmatrix}\right)$. You have to choose two more vectors $v_3,v_4$, such that $\{v_1,v_2,v_3,v_4\}$ is a basis of $\R^{4}$. One possible choice would be $v_3 = \left(\begin{smallmatrix}0\\0\\1\\ 0\end{smallmatrix}\right)$ and $v_4 = \left(\begin{smallmatrix}0\\0\\0\\ 1\end{smallmatrix}\right)$. Then you choose that $f$ maps $$ v_3 \mapsto \left(\begin{smallmatrix}1\\1\\2\end{smallmatrix}\right), \quad v_4 \mapsto \left(\begin{smallmatrix}7\\5\\0\end{smallmatrix}\right). $$ Now you just have to calculate the image of $e_1 (=\left(\begin{smallmatrix}1\\0\\0\\0\end{smallmatrix}\right))$ and $e_2 (=\left(\begin{smallmatrix}0\\1\\0\\0\end{smallmatrix}\right))$. \begin{align} \left(\begin{matrix} 1 & 1 & 0 & 0 \\ 2 & 5 & 0 & 0 \\ 3 & 4 & 1 & 0 \\ 1 & 1 & 0 & 1 \\ \hline 1 & 0 & 1 & 7 \\ 3 & 0 & 1 & 5 \\ 1 & 0 & 2 & 0 \\ \end{matrix}\right) \rightsquigarrow \text{many elementary steps} \rightsquigarrow \left(\begin{matrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \hline -\frac{23}{3} & -\frac{2}{3} & 1 & 7 \\ -\frac{7}{3} & -\frac{4}{3} & 1 & 5 \\ -3 & -1 & 2 & 0 \\ \end{matrix}\right) \end{align} Therefore, the matrix which represents your linear mapping $f$ is $$ \left( \begin{matrix} -\frac{23}{3} & -\frac{2}{3} & 1 & 7 \\ -\frac{7}{3} & -\frac{4}{3} & 1 & 5 \\ -3 & -1 & 2 & 0 \\ \end{matrix}\right). $$

Matrices are just linear maps, and so are defined by where they send the elements of any given basis. If we construct a basis out of $(1,2,3,1)$, $(1,5,4,1)$, and any two vectors that form, along with those two, a linearly independent set, then we just need to send everything to the right places, and we're done (we can do this because $(1,2,3,1)$ and $(1,5,4,1)$ are linearly independent, otherwise this wouldn't be possible in general). So, let's start by picking our other two basis vectors: this is essentially arbitrary, but $(1,0,0,0)$ and $(0,0,0,1)$ work, so we'll use those. Thus, we can just choose our linear map to be the one that sends $(1,2,3,1)$ to $(1,3,1)$, $(1,5,4,1)$ to $(0,0,0)$, $(1,0,0,0)$ to $(1,1,2)$, and $(0,0,0,1)$ to $(7,5,0)$. This is well-defined because our basis is linearly independent, and uniquely defines a linear map because it spans.

Now, to construct a matrix for our linear map, we simply find the images of the standard basis vectors: for two of them, it's really easy, because we've already defined their images: $f(1,0,0,0) = (1,1,2)$, and $f(0,0,0,1) = (7,5,0)$. For the other two, we first need to write them in terms of our basis vectors: it turns out that $\frac{1}{7}(-4(1,2,3,1) + 3(1,5,4,1) + (1,0,0,0) + (0,0,0,1)) = (0,1,0,0)$, and $\frac{1}{7}(5(1,2,3,1) - 2(1,5,4,1) - 3(1,0,0,0) - 3(0,0,0,1)) = (0,0,1,0)$, so \begin{align*}f(0,1,0,0) &= \frac{1}{7}(-4f(1,2,3,1) + 3f(1,5,4,1) + f(1,0,0,0) + f(0,0,0,1)) \\&=\frac{1}{7}(-4(1,3,1) + 3(0,0,0) + (1,1,2) + (7,5,0)) \\&=\left(\frac{4}{7},\frac{-6}{7},\frac{-2}{7}\right),\end{align*} and \begin{align*}f(0,0,1,0) &= \frac{1}{7}(5f(1,2,3,1) - 2f(1,5,4,1) - 3f(1,0,0,0) - 3f(0,0,0,1)) \\&= \frac{1}{7}\left(5(1,3,1) - 2(0,0,0) + (1,1,2) + (7,5,0)\right) \\&= \left(\frac{13}{7},3,1\right).\end{align*}

We can now assemble these into our matrix: our linear map has matrix, with respect to the standard basis

$$\left(\array{1&\frac{4}{7}&\frac{13}{7}&7\\1&\frac{-6}{7}&3&5\\2&\frac{-2}{7}&1&0}\right).$$

[NB: I've assumed you're writing your matrices on the left, despite having used row vectors. If my assumption is incorrect, transpose this matrix]

As you may have noticed, this isn't unique. To finish things off with the analytic formula:

$$f(x,y,z,w) = \left(\array{1&\frac{4}{7}&\frac{13}{7}&7\\1&\frac{-6}{7}&3&5\\2&\frac{-2}{7}&1&0}\right)\left(\array{x\\y\\z\\w}\right) = \left(\array{x+\frac{4y}{7}+\frac{13z}{7}+7w\\x-\frac{6y}{7} + 3z + 5w\\2x -\frac{2y}{7}+z}\right).$$