How to change the vertical position of a horizontal line in a matrix?

With a trick

\documentclass[11p]{book}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{mathtools}

\begin{document}
    \begin{equation}
        R = \left[ \begin{array}{ccc|ccc}
        A & B & C & A & B &C \\[2ex]
        A & B & C & A & B &C \\[2ex]
        A & B & C & A & B &C \\[.65ex]
        \hline
        \rule[0pt]{2pt}{\heightof{A}+1ex}A & B & C & A & B &C \\[2ex]
        A & B & C & A & B &C \\[2ex]
        A & B & C & A & B &C
    \end{array} \right]
    \end{equation}
\end{document}

Replace \rule[0pt]{2pt}{\heightof{A}+1ex} by \rule[0pt]{0pt}{\heightof{A}+1ex}. Also the [.65ex] could certainly be computed in a better way by getting the interline value in the matrix when the option \\[0ex] is used. I do not know this value.

For an exact control, you could go with

\documentclass[11p]{book}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{mathtools}
\begin{document}
    \begin{equation}%
    \renewcommand{\arraystretch}{0}%
        R = \left[ \begin{array}{ccc|ccc}
        A & B & C & A & B &C \\[2ex]
        A & B & C & A & B &C \\[2ex]
        A & B & C & A & B &C \\[1ex]
        \hline
        \rule[0pt]{2pt}{\heightof{A}+1ex}%
        A & B & C & A & B &C \\[2ex]
        A & B & C & A & B &C \\[2ex]
        A & B & C & A & B &C
    \end{array} \right]
    \end{equation}

Another possibility, withcellspace:

\documentclass[11pt, letterpaper, twoside]{book}
\usepackage[T1]{fontenc}
\usepackage{lmodern}
\usepackage[total={6in,10in},left=1.5in,top=0.5in,includehead,includefoot]{geometry}
\usepackage{amsfonts}
\usepackage{mathtools}

\usepackage[math]{cellspace}
\setlength{\cellspacetoplimit}{1.6ex}
\setlength{\cellspacebottomlimit}{1.6ex}

\begin{document}

\begin{equation}
    R = \left[ \begin{array}{ccc|>{$}Sc<{$}cc}
    \multicolumn{6}{c} {}\\
    \noalign{\vspace{-3.2ex}}
    A & B & C & A & B &C \\
    A & B & C & A & B &C \\
    A & B & C & A & B &C \\
    \hline
    A & B & C & A & B &C \\
    A & B & C & A & B &C \\
    A & B & C & A & B &C \\[-0.6ex]
\end{array} \right]. 
\end{equation}

\end{document} 

I think that this code it is possible to create, quickly, with nicematrix package. Here I add my humble version using array enviroment.

\documentclass[a4paper,12pt]{article}
\usepackage{amsmath}

\begin{document}
$\left(\begin{array}{@{}c|c@{}}
  \begin{matrix}
  A & B & C\\
  A & B & C\\
  A & B & C
  \end{matrix}
  & \begin{matrix}
  A & B & C\\
  A & B & C\\
  A & B & C
  \end{matrix} \\
\hline
  \begin{matrix}
  A & B & C\\
  A & B & C\\
  A & B & C
  \end{matrix}&
  \begin{matrix}
  A & B & C\\
  A & B & C\\
  A & B & C
  \end{matrix}
\end{array}\right)$
\end{document}

Or this rudimental solution with the vertical space for the matrices [.3cm].

\documentclass[a4paper,12pt]{article}
\usepackage{amsmath}

\begin{document}
$R=\left(\begin{array}{@{}c|c@{}}
  \begin{matrix}
  A & B & C\\[.3cm]
  A & B & C\\[.3cm]
  A & B & C
  \end{matrix}
  & \begin{matrix}
  A & B & C\\[.3cm]
  A & B & C\\[.3cm]
  A & B & C
  \end{matrix}\\[.1cm] \\[-.1cm]
\hline\\
  \begin{matrix}
  A & B & C\\[.3cm]
  A & B & C\\[.3cm]
  A & B & C
  \end{matrix}&
  \begin{matrix}
  A & B & C\\[.3cm]
  A & B & C\\[.3cm]
  A & B & C
  \end{matrix}
\end{array}\right).$
\end{document}