How to draw a diagonal rectangle with TikZ?

This answer shows how to get a rotated rectangle to fit around any two given nodes, so does not require any change to the way you set-up the picture.

\documentclass{article}
\usepackage{amsmath, amsfonts, amssymb}
\usepackage{tikz}
\usetikzlibrary{fit}

\begin{document}
    \begin{tikzpicture}
        \node (a) at (0, 0) {a};
        \node (b) at (1,0) {b};
        \node (c) at (2,0) {c};
        \node (d) at (3,0) {d};

        \node (e) at (0,-0.5) {e};
        \node (f) at (1,-0.5) {f};
        \node (g) at (2,-0.5) {g};
        \node (h) at (3,-0.5) {h};

        \node (i) at (0,-1) {i};
        \node (j) at (1,-1) {j};
        \node (k) at (2,-1) {k};
        \node (l) at (3,-1) {l};

        \node (m) at (0,-1.5) {m};
        \node (n) at (1,-1.5) {n};
        \node (o) at (2,-1.5) {o};
        \node (p) at (3,-1.5) {p};
        \pgfmathanglebetweenpoints{\pgfpointanchor{a}{center}}{\pgfpointanchor{p}{center}}
        \pgfmathsetmacro{\myangle}{\pgfmathresult}
        \node[draw=blue, rounded corners=2pt, rotate fit=\myangle, fit=(a) (p)] {};
    \end{tikzpicture}
\end{document}

The main idea is to use the fit library, but you need to add a calculation of the rotation angle. Without an angle, fit just produces a rectangle containing the nodes, but with edges parallel to the coordinate axes.

\documentclass[border=2mm]{standalone}
\usepackage{tikz}
\usetikzlibrary{matrix,calc}
\begin{document}
\begin{tikzpicture}
  \matrix (M) [matrix of nodes]
    { a & b & c & d \\
      e & f & g & h \\
      i & j & k & l \\
      m & n & o & p \\
    };
  \draw[blue,rounded corners]
    let \p1=($(M-1-1)!-2mm!(M-4-4)$),
        \p2=($(M-4-4)!-2mm!(M-1-1)$),
        \p3=($(\p1)!2mm!90:(\p2)$),
        \p4=($(\p1)!2mm!-90:(\p2)$),
        \p5=($(\p2)!2mm!90:(\p1)$),
        \p6=($(\p2)!2mm!-90:(\p1)$)
    in
    (\p3) -- (\p4)-- (\p5) -- (\p6) -- cycle;
\end{tikzpicture}
\end{document}

For your MWE that came a bit late:

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{calc}
\begin{document}
    \begin{tikzpicture}
        \node (a) at (0, 0) {a};
        \node (b) at (1,0) {b};
        \node (c) at (2,0) {c};
        \node (d) at (3,0) {d};

        \node (e) at (0,-0.5) {e};
        \node (f) at (1,-0.5) {f};
        \node (g) at (2,-0.5) {g};
        \node (h) at (3,-0.5) {h};

        \node (i) at (0,-1) {i};
        \node (j) at (1,-1) {j};
        \node (k) at (2,-1) {k};
        \node (l) at (3,-1) {l};

        \node (m) at (0,-1.5) {m};
        \node (n) at (1,-1.5) {n};
        \node (o) at (2,-1.5) {o};
        \node (p) at (3,-1.5) {p};
  \draw[blue,rounded corners]
    let \p1=($(a)!-2mm!(p)$),
        \p2=($(p)!-2mm!(a)$),
        \p3=($(\p1)!2mm!90:(\p2)$),
        \p4=($(\p1)!2mm!-90:(\p2)$),
        \p5=($(\p2)!2mm!90:(\p1)$),
        \p6=($(\p2)!2mm!-90:(\p1)$)
    in
    (\p3) -- (\p4)-- (\p5) -- (\p6) -- cycle;
    \end{tikzpicture}
\end{document}