Defining a new unit in siunitx that has a subscript

Remove the spaces from the definition. The reason why the space doesn't occur for \electronmass is I'd think that siunitx is an expl3 package, and in the environment defined by expl3 spaces in the input are all ignored. Your new definition is not written in that environment.

\documentclass{article}

\usepackage{siunitx}


\DeclareSIUnit[]\protonmass
{\text{\ensuremath{m_{\textup{p}}}}}

\begin{document}

electron mass = \si{\electronmass}

proton mass = \si{\protonmass}

\end{document}

Here's the output.

After the definition

\DeclareSIUnit[]\protonmass
{ \text { \ensuremath { m _ { \textup { p } } } } }

the replacement text for \protonmass becomes

•\text{•\ensuremath{•m•_{•\textup{•p•}•}•}•}•

where • denotes a space token. The third, fourth, fifth, eighth and ninth space tokens get ignored, because they happen to be evaluated in math mode. The remaining ones aren't (but the outer ones are removed by siunitx) because they're in text mode. So you still get the second, sixth, seventh and tenth space tokens: a total of four. Indeed

\documentclass{article}

\usepackage{siunitx}


\DeclareSIUnit[]\protonmass
{ \text { \ensuremath { m _ { \textup { p } } } } }

\begin{document}

X\si{\electronmass}X

X\si{\protonmass}X

\end{document}

produces

where the four spaces are clearly visible.

Correct definition

\documentclass{article}

\usepackage{siunitx}

\DeclareSIUnit{\protonmass}{%
 \text{$m_{\textup{p}}$}%
}

\begin{document}

X\si{\electronmass}X

X\si{\protonmass}X

\end{document}

Notes

1. \ensuremath is not needed, because you know to be in text mode
2. Joseph Wright, the package author, should provide the unit definitions in a different file not under \ExplSyntaxOn, in order to provide a model for further definitions.