Matrices and TikZ : positioning a table
I would do this way.
Please note that if you set every left delimiter/.append style={name=rd} and the tikzpicture level, every left delimiter node would have the same name.
If inner xsep=inner ysep you could simply use inner sep.
Moreover, you don't need a tabular in the node text, you can use align=....
Also, please note the use of midway, anchor, and right/left/above/below = ... of ..." (librarypositioning`) instead of fixed cooordinates.
\documentclass[border=10pt]{standalone}
\usepackage{tikz}
\usetikzlibrary{matrix,arrows.meta,fit,backgrounds,positioning}
\tikzset{myyellow/.style={
fill=yellow!90, draw opacity=0,
draw=yellow!90, fill opacity=0.3,
}}
\begin{document}
\begin{tikzpicture}[thick,baseline=(A.center),>=Stealth]
\begin{scope}[every left delimiter/.append style={name=rdA}]
\matrix[matrix of math nodes,left delimiter=(,right delimiter=),
column sep=20pt] (A) at (0,0) {
a_{1,1} & a_{1,2} & \dots & a_{1,j} & \dots & a_{1,n} \\
a_{2,1} & a_{2,2} & \dots & a_{2,j} & \dots & a_{2,n} \\
\vdots & & & \vdots & & \vdots \\
|[red]|a_{i,1} & |[green]|a_{i,2} & \dots & |[blue]|a_{i,j} & \dots & |[violet]|a_{i,n} \\
\vdots & & & \vdots & & \vdots \\
a_{n,1} & a_{n,2} & \dots & a_{n,j} & \dots & a_{n,n} \\
};
\end{scope}
\matrix [matrix of math nodes,left delimiter=(,right delimiter=), right=20pt of A] (P) {
p_1 \\
p_2 \\
\vdots \\
p_i=\textcolor{red}{a_{i,1}}\times\textcolor{red}{b_1}+\textcolor{green} {a_{i,2}}\times \textcolor{green}{b_2}+ \dots + \textcolor{violet}{a_{i,n}}\times \textcolor{violet}{b_n} \\
\vdots \\
p_n \\
};
\begin{scope}[every left delimiter/.append style={name=rdB}]
\matrix [matrix of math nodes,left delimiter=(,right delimiter=), above=4pt of P.north] (B) {
|[red]|b_1\\
|[green]|b_2\\
\vdots\\
|[blue]|b_j\\
\vdots\\
|[violet]|b_n\\
};
\end{scope}
\begin{scope}[on background layer]
\node [fit=(A-4-1) (A-4-6),myyellow,inner sep=0pt]{};
\end{scope}
\node[left =-2pt of rdA] {$A=$};
\node[left =-2pt of rdB] {$B=$};
\draw[<->,red](A-4-1.east) -- ++(10pt,0) |- (rdB.west|-B-1-1.center);
\draw[<->,green](A-4-2.east) -- ++(10pt,0) |-(rdB.west|-B-2-1.center) node[midway, below right, align=left, black, font=\itshape] {on multiplie\\
ces coefficients};
\draw[<->,blue](A-4-4.east) -- ++(10pt,0) |- (rdB.west|-B-4-1.center);
\draw[<->,violet](A-4-6.west) -- ++ (-10pt,0) |- (rdB.west|-B-6-1.center);
\node[font=\itshape, anchor=north west, align=left] at (P.north west){
on ajoute\\
les produits
};
\end{tikzpicture}
\end{document}
If your question is whether or not it is possible to get rid of all the shifts, the answer is yes.
\documentclass[border=10pt]{standalone}
\usepackage{tikz}
\usetikzlibrary{matrix,decorations.pathreplacing,calc,fit,backgrounds,positioning}
\tikzset{myyellow/.style={
fill=yellow!90,fill opacity=0.3}}
\begin{document}
\begin{tikzpicture}[thick,baseline=(A.center),>=stealth,every left delimiter/.append style={name=rd},]
\matrix [matrix of math nodes,left delimiter=(,right delimiter=)] (A) {
a_{1,1} & a_{1,2} & \dots & a_{1,j} & \dots & a_{1,n} \\
a_{2,1} & a_{2,2} & \dots & a_{2,j} & \dots & a_{2,n} \\
\vdots & & & \vdots & & \vdots \\
|[red]|a_{i,1} & |[green]|a_{i,2} & \dots & |[blue]|a_{i,j} & \dots & |[violet]|a_{i,n} \\
\vdots & & & \vdots & & \vdots \\
a_{n,1} & a_{n,2} & \dots & a_{n,j} & \dots & a_{n,n} \\
};
\matrix [matrix of math nodes,left delimiter=(,right delimiter=),right=2em of A]
(P) {
p_1 \\
p_2 \\
\vdots \\
p_i=\textcolor{red}{a_{i,1}}\times\textcolor{red}{b_1}+\textcolor{green} {a_{i,2}}\times \textcolor{green}{b_2}+ \dots + \textcolor{violet}{a_{i,n}}\times \textcolor{violet}{b_n} \\
\vdots \\
p_n \\
};
\matrix [matrix of math nodes,left delimiter=(,right delimiter=),above=2ex of P] (B) {
|[red]|b_1\\
|[green]|b_2\\
\vdots\\
|[blue]|b_j\\
\vdots\\
|[violet]|b_n\\
};
\begin{scope}[on background layer]
\node [fit=(A-4-1) (A-4-6),myyellow,inner sep=0pt]{};
\end{scope}
\draw[<->,red](A-4-1) --(A-4-1.east |- B-1-1.center)--(rd.west|-B-1-1.center);
\draw[<->,green](A-4-2)--(A-4-2.east |- B-2-1.center)--(rd.west|-B-2-1.center);
\draw[<->,blue](A-4-4)-- (A-4-4.east |- B-4-1.center)--(rd.west|-B-4-1.center) ;
\draw[<->,violet](A-4-6)-- (A-4-6.north east |- B-6-1.center)-- (rd.west|-B-6-1.center) ;
\node[anchor=north west,font=\itshape,align=left,inner ysep=2ex] at (P.north west){
on ajoute\\
les produits};
\node[anchor=north west,font=\itshape,align=left] at (A-4-2.east |- B-2-1.center){
on multiplie\\
ces coefficients\\
};
\node[left=1em of A]{$A=$};
\node[left=1em of B]{$B=$};
\end{tikzpicture}
\end{document}
I do, however, not necessarily regard the shifts you are employing as inelegant. After all, you were positioning everything relative to each other. That is, your code is IMHO quite elegant to begin with.
This solution puts the midpoint on a line 80 degrees up from the start. The code is somewhat cumbersome, but the results are IMHO prettier.
\documentclass[border=10pt]{standalone}
\usepackage{tikz}
\usetikzlibrary{matrix,decorations.pathreplacing,calc,fit,backgrounds,positioning,intersections}
\tikzset{myyellow/.style={
fill=yellow!90, draw opacity=0,
draw=yellow!90, fill opacity=0.3,
}}
\begin{document}
\begin{tikzpicture}[thick,baseline=(A.center),>=stealth,every left delimiter/.append style={name=rd},]
\matrix [matrix of math nodes,left delimiter=(,right delimiter=)] (A) at (0,0) {
a_{1,1} & a_{1,2} & \dots & a_{1,j} & \dots & a_{1,n} \\
a_{2,1} & a_{2,2} & \dots & a_{2,j} & \dots & a_{2,n} \\
\vdots & & & \vdots & & \vdots \\
|[red]|a_{i,1} & |[green]|a_{i,2} & \dots & |[blue]|a_{i,j} & \dots & |[violet]|a_{i,n} \\
\vdots & & & \vdots & & \vdots \\
a_{n,1} & a_{n,2} & \dots & a_{n,j} & \dots & a_{n,n} \\
};
\matrix [matrix of math nodes,left delimiter=(,right delimiter=)] (P) at (6.5,0) {
p_1 \\
p_2 \\
\vdots \\
p_i=\textcolor{red}{a_{i,1}}\times\textcolor{red}{b_1}+\textcolor{green} {a_{i,2}}\times \textcolor{green}{b_2}+ \dots + \textcolor{violet}{a_{i,n}}\times \textcolor{violet}{b_n} \\
\vdots \\
p_n \\
};
\matrix [matrix of math nodes,left delimiter=(,right delimiter=),yshift=2cm] (B) at (P.north) {
|[red]|b_1\\
|[green]|b_2\\
\vdots\\
|[blue]|b_j\\
\vdots\\
|[violet]|b_n\\
};
\begin{scope}[on background layer]
\node [fit=(A-4-1) (A-4-6),myyellow,inner xsep=0pt,inner ysep=0pt]{};
\end{scope}
\begin{scope}% safety
\clip (current bounding box.south west) rectangle (current bounding box.north east);
\path[name path=a1] (A-4-1) -- ++(80: 10);
\path[name path=b1] (B-1-1.center) -- ++(180:10);
\path[name intersections={of=a1 and b1}]
(intersection-1) coordinate (c1);
\path[name path=a2] (A-4-2) -- ++(80: 10);
\path[name path=b2] (B-2-1.center) -- ++(180:10);
\path[name intersections={of=a2 and b2}]
(intersection-1) coordinate (c2);
\path[name path=a3] (A-4-4) -- ++(80: 10);
\path[name path=b3] (B-4-1.center) -- ++(180:10);
\path[name intersections={of=a3 and b3}]
(intersection-1) coordinate (c3);
\path[name path=a4] (A-4-6) -- ++(80: 10);
\path[name path=b4] (B-6-1.center) -- ++(180:10);
\path[name intersections={of=a4 and b4}]
(intersection-1) coordinate (c4);
\end{scope}
\draw[<->,red](A-4-1) --(c1)--(rd.west|-B-1-1.center);
\draw[<->,green](A-4-2)--(c2)--(rd.west|-B-2-1.center);
\draw[<->,blue](A-4-4)-- (c3)--(rd.west|-B-4-1.center) ;
\draw[<->,violet](A-4-6)-- (c4)-- (rd.west|-B-6-1.center);
\node[xshift=-2cm,font=\it] at (P-2-1.west){
\begin{tabular}{l}
on ajoute\\
les produits
\end{tabular}
};
\node[above,font=\it] at (c3){
\begin{tabular}{l}
on multiplie\\
ces coefficients\\
\end{tabular}
};
\node[xshift=-0.7cm] at (A.west){$A=$};
\node[xshift=-0.7cm] at (B.west){$B=$};
\end{tikzpicture}
\end{document}