Possible to define new unit, comparable to pt or em?

As the comments state the units that are know to TeX are implemented in the Pascal source. But macros can be written to do what you want as Joseph Wright points out in his comment.

Two assumptions:

1. Use for the unit two characters as it is done by TeX for all other units. (It avoids a lot of tests.)
2. Use the new unit always as a macro parameter in curly braces.

I show the general approach for two macros \Setlength and \Hspace, which are mentioned in your question and your comments. Note I do not change the LaTeX macros.

I write the macros in plain TeX style. The macros do not check if the unit is a known unit like pt. So they can only be used with your own units. Moreover they do not give useful error messages if they are called with an unknown unit. As I wrote I only give the general approach to attack the problem.

\newdimen\ownunitdimen

\def\rev#1#2\rev{\ifx\rev#2\rev#1\else\rev#2\rev#1\fi}% a TeX pearl
\def\applyownunit#1#2#3\done{% needs \#2#1factor and \#2#1unit
   \global\ownunitdimen=\rev #3\rev\csname #2#1unit\endcsname
   \global\ownunitdimen=\csname #2#1factor\endcsname\ownunitdimen\ignorespaces}

\def\Setlength#1#2{\bgroup\edef\reverse{\rev #2\rev}%
   \expandafter\applyownunit\reverse\done\egroup
   \csname #1\endcsname=\ownunitdimen}
\def\Hspace#1{\bgroup\edef\reverse{\rev #1\rev}%
   \expandafter\applyownunit\reverse\done\egroup
   \hskip\ownunitdimen\relax}

% define new units; avoid dimen registers
\def\dmfactor{10}\def\dmunit{cm}% 1 dm = 10 cm
\def\pzfactor{2.2633484517438}\def\pzunit{mm} % 1 pz \approx 2.2633484517438 mm
% see page 26 of ``Selected Papers on Fun \& Games'' by Donald E. Knuth

% test
\newdimen\A \newdimen\B
\Setlength{A}{8.97dm}% use the new unit
\B=89.7cm %      and compare it to this

A: \the\A \Hspace{5,2pz}B: \the\B\par % use the new unit in European style
A: \the\A \hskip11.76941194906776mm B: \the\B % and compare

% I think this is better
\newdimen\pz \pz=2.2633484517438mm
A: \the\A \hskip5.2\pz B: \the\B

\bye