Define a command differently in inline vs display mode

Since you are using LuaTeX, you can simply query \mathstyle.

\documentclass{article}
\usepackage{amsmath}
\usepackage{unicode-math}

\newcommand\catlim{%
    \ifcase\mathstyle
        % 0 = \displaystyle
        \expandafter\varprojlim
    \or
        % 1 = \crampeddisplaystyle
        \expandafter\varprojlim
    \else
        % all other styles
        \expandafter\lim
    \fi}

\begin{document}

Inline \(\catlim_{i \in \mathbb{N}} \mathbb{Z} / p^i \mathbb{Z}\) and display:
\begin{equation*}
  \catlim_{i \in \mathbb{N}} \mathbb{Z} / p^i \mathbb{Z}.
\end{equation*}

\end{document}

It’s not clear what you want to do when you explicitly reset the math style. The following solution, which requires amsmath but works also with the “classical” typesetting engines, discriminates based on the \if@display switch. This purposely ignores the math style (\displaystyle vs. other styles), and decides which command to use (\varprojlim vs. \lim) considering only if the formula is being displayed or typeset in-line.

I insist that it is not clear from your question whether you consider this a bug or a feature. See also @daleif’s comment.

The code:

% My standard header for TeX.SX answers:
\documentclass[a4paper]{article} % To avoid confusion, let us explicitly 
                                 % declare the paper format.

\usepackage[T1]{fontenc}         % Not always necessary, but recommended.
% End of standard header.  What follows pertains to the problem at hand.

\usepackage{mathtools} % also loads "amsmath"
\usepackage{amsfonts}

\makeatletter

\newcommand*\catlim{%
  \relax\if@display
    \expandafter\varprojlim
  \else
    \expandafter\lim
  \fi
}

\makeatother

\newcommand*{\numberset}[1]{\mathbb{#1}}
\newcommand*{\N}{\numberset{N}}
\newcommand*{\Z}{\numberset{Z}}



\begin{document}

In line:
\( \catlim_{i\in\N} \Z/p^{i}\Z \).

In line, but with \verb|\displaystyle|:
\( \displaystyle \catlim_{i\in\N} \frac{\Z}{p^{i}\Z} \).

In display:
\[
    \catlim_{i\in\N} \frac{\Z}{p^{i}\Z}
\]

In display, but with \verb|\textstyle|:
\[
    \textstyle \catlim_{i\in\N} \Z/p^{i}\Z
\]

An example with \texttt{cases} (starred version requires the 
\textsf{mathtools} package):
\[
    G =
        \begin{cases*}
            \catlim_{i\in\N} \frac{\Z}{p^{i}\Z} & first case \\
            \catlim_{i\in\N} \frac{\Z}{q^{i}\Z} & second case
        \end{cases*}
\]
Note how the fraction is typeset: this too shows that
\verb|\textstyle| is in force.

\end{document}

The printout: