Decrease font size by one point

The current font size is stored in the macro \f@size:

\RequirePackage{fix-cm} % or use a scalable font
\documentclass{article}

\makeatletter
\newcommand{\oneptsmaller}[1]{%
  \begingroup
  \fontsize{\dimexpr\f@size pt-1pt}{\f@baselineskip}\selectfont
  #1%
  \endgroup
}
\makeatother

\begin{document}

\fontname\font\ \oneptsmaller{\fontname\font}

\large
\fontname\font\ \oneptsmaller{\fontname\font}

\footnotesize
\fontname\font\ \oneptsmaller{\fontname\font}

\end{document}

With a scalable font, say \usepackage{baskervald},

Note: \fontname\font is used just to show what font is currently used. In the case of ybvr8t there is no at clause because the font is at its natural size of 10pt.

You can also use the package relsize, which allows for what you need. Consider this MWE:

\documentclass{article}
\usepackage{relsize}
\begin{document}
This is normal size font.

\smaller

And this is one point smaller.
\end{document}