Any element $g$ of $GL(2,p)$ of order $p$, $p$ prime, is conjugate to $\begin{bmatrix}1&1\\0&1\end{bmatrix}$

Here is a solution using linear algebra over $\mathbb F_p$.

$g$ satisfies $0=g^p-I=(g-I)^p$. Therefore, $1$ is the only eigenvalue of $g$. Since $g$ is a $2\times 2$ matrix, these are the only possible Jordan forms for $g$: $$ \begin{bmatrix}1&0\\0&1\end{bmatrix} \quad\text{and}\quad \begin{bmatrix}1&1\\0&1\end{bmatrix} $$ But $I$ does not have order $p$.

It sounds like you want a mixture of group actions and linear algebra.

---

Let $g\in G=GL_2(\Bbb{F}_p)$ be an element of order $p$. Let's denote $H=\langle g\rangle$, and $X=\Bbb{F}_p^2$ the set of (column) all vectors on which both $H$ and $G$ act by matrix multiplication from the left.

1. By the orbit-stabilizer theorem the orbits of $H$ have sizes $1$ and $p$ only.
2. Because the zero vector forms an $H$-orbit of size $1$ and $|X|=p^2$, there must be other $H$-orbits of size $1$. Let $x\in X$ form such a singleton orbit of $H$.
3. The scalar multiples of $x$ are then all eigenvectors of $g$ belonging to eigenvalue $\lambda=1$. Let $y\in X$ be a vector that is linearly independent from $x$. Then $y$ cannot belong to the eigenvalue $1$ for then we would have $g=1_G$.
4. The vectors $x$ and $y$ form a basis $\mathcal{B}$ of $X$ over $\Bbb{F}_p$, so we have $g\cdot x=x$ and $g\cdot y=ax+by$ for some $a,b\in\Bbb{F}_p$. The matrix of $g$ with respect to $\mathcal{B}$ looks like $$ M_{\mathcal{B}}(g)=\left(\begin{array}{cc} 1&a\\ 0&b \end{array}\right). $$
5. We have $\det(g)=b$, so $1=\det(g^p)=b^p=b$.
6. Because $g\neq 1_G$ we have $a\neq0$.
7. With respect to the basis $\mathcal{B}'=\{ax,y\}$ the matrix of $g$ thus looks like $$ M_{\mathcal{B}'}(g)=\left(\begin{array}{cc} 1&1\\ 0&1 \end{array}\right). $$