Intersections in Tikz

it is not clear (at least to me) where you like to have your highlighted rectangle. you not draw perpendicular lines to vector S nor to vector .x_3 ...

edit: after your edit of your question it seems that you looking for the following:

\documentclass[12pt,a4paper]{article}
\usepackage{tikz}
\usetikzlibrary{angles, calc}

\begin{document}
    \begin{tikzpicture}
% axes
\coordinate (A) at (0,0);
\draw[thick,->] (A) -- (4.5,0) node[below right, node font=\small] {Security 1};
\draw[thick,->] (A) -- (0,4.5) node[above  left, node font=\small] {Security 2};
% auxilary lines
\foreach \alpha in {-15,105}
{
\draw  (A) -- (\alpha:3.5);
}
% vectors,
\foreach \alpha [count=\i] in {15,45,75}
{\draw[->] (0,0) -- (\alpha:3.5) coordinate (x\i);
 \path (x\i) ++ (\alpha:0.3) node {$x_{.\i}$};
}
% blue vector
\coordinate (B) at (0.4,3.5);
\draw [blue, thick,->] (A) -- (B) node[pos=1.08, font=\small] {$S$};
% shadows 
\draw[red, text=black]  
    (A) -- ($(A)!3.5cm!270:(B)$)  coordinate (x)  % perpendicular to S
    (A) -- ($(A)!3.5cm!270:(x3)$) coordinate (y)  % perpendicular to .x_3
    (A) -- ($(A)!2.5cm! 90:(B)$)  coordinate[pos=0.5] (z); % perpendicular to S
\fill[blue!70] (z) |- (x) -- cycle;
\fill[blue!30] (x) -- (y) -- (A);
% angles, in case that you like to have
\path pic [draw,angle radius = 9mm] {angle = x--A--B}
      pic [draw,angle radius =11mm] {angle = B--A--z}
      pic [draw,angle radius =13mm] {angle = y--A--x3};
  \end{tikzpicture}
\end{document}

Something like this? (I shaded the area in blue.)

\documentclass[12pt,a4paper]{article}
\usepackage{tikz}
\usetikzlibrary{calc}
\begin{document}
\begin{tikzpicture}[dot/.style={circle,inner sep=1pt,fill,label={#1},name=#1},
  extended line/.style={shorten >=-#1,shorten <=-#1},
  extended line/.default=1cm]

 \draw[thick,->] (0,0) -- (4.5,0) node[anchor=west,font=\small] {Security 1};
 \draw[thick,->] (0,0) -- (0,4.5) node[anchor=south,font=\small] {Security 2};

 \coordinate (A) at (0,0);
 \coordinate (B) at (0.4,3.5);

 \draw ($(A)!4cm!270:(B)$) -- ($(A)!2cm!90:(B)$)  coordinate[pos=0.75] (x) 
 coordinate[pos=0.1] (y); %specify a further coordinate to include additional vectors
 \draw[fill](x)--(y)-| cycle; [% filled triangle]

  \foreach \x [count=\xi] in {15,45,75} { %alternative way
   \draw[->](0,0)--(\x:3.5) coordinate (\xi); 
   \node at (\x:3.9){$x_{.\xi}$}; 
   \draw (A) -- ($(A)!{sign(45-\x)*3.5cm}!90:(\xi)$) coordinate(p\xi); 
 }

 \draw [blue, thick,->] (A) -- (B) 
 node[pos=1.04,above, font=\small]{$S$};  
 \draw (-15:1) arc (-15:105:1);    
 \fill[blue] (A) -- (p3) --  ($(A)!(p3)!(y)$) coordinate (aux)-- cycle;
 \draw[latex-] ($($(p3)!0.6!(aux)$)+(-0.3,0)$) -- ++(1.2,0) 
 node[right,font=\small] {arbitragage portfolios};
\end{tikzpicture}
\end{document}