How to align braces and \mid in a set in order to get the same height

I suggest using the pmatrix*[r] environment, from mathtools, in a medium-sized version thanks to the medsize environment from nccmath. Also, I used a \set command based on \DeclarePairedDelimiterX (again from mathtools), which uses two arguments, separated by a semi-colon:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amssymb}
\usepackage{mathtools, nccmath}

\newenvironment{pmmatrix*}[1][c]{\medsize\begin{pmatrix*}[#1]}{ \end{pmatrix*}\endmedsize}
\usepackage{xparse}
\DeclarePairedDelimiterX{\set}[1]\{\}{\setargs{#1}}
\NewDocumentCommand{\setargs}{>{\SplitArgument{1}{;}}m}
{\setargsaux#1}
\NewDocumentCommand{\setargsaux}{mm}
{\IfNoValueTF{#2}{#1}{\nonscript\,#1\nonscript\;\delimsize\vert\nonscript\:\allowbreak\mathopen{}#2\nonscript\,}}

\begin{document}

\[\set*{\begin{pmmatrix*}[r]1\\ -2 \\ 3 \\ 4 \end{pmmatrix*} + \lambda\begin{pmmatrix*}[r]-2 \\ 3 \\ 4 \\ -5 \end{pmmatrix*}; \lambda\in\mathbf{R}} \]

\end{document} 

Using spalign package you can obtain same result of the best answers of @Bernard and @Steven B. Segletes. The code is not very long.

\documentclass[12pt]{article}
\usepackage{spalign}
\usepackage{amssymb}

\begin{document}

\[G_1=\left\{ \spalignvector[r]{1 -2 3 4}+\lambda\spalignvector[r]{1 -2 3 4} \middle\vert\ \lambda\in \mathbb{R}\right\}\]
\end{document}

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{tabstackengine}
\begin{document}

$G_1=\left\{ \parenVectorstack[r]{ 1 \\ -2 \\ 3 \\ 4 } + 
  \lambda \parenVectorstack[r]{ -2 \\ 3 \\ 4 \\ -5 } 
  \middle\vert\, \lambda \in \mathbb{R}\right\}$
\end{document}