Command doing the same job as StrBefore but being expandable

One approach is using the optional argument of StrBefore:

\documentclass{book}
\usepackage{xstring}
\usepackage{ifthen}

\newcounter{try}

\newcommand{\foo}[1]{\StrBefore{#1}{.}[\result]}

\newcommand\test[1]{\foo{#1}\setcounter{try}{\result}\arabic{try}}%


\begin{document}

Result of foo\{3.2\}: \foo{3.2}
Result of test\{3.2\}: \test{3.2}

\end{document}

Another approach would be to parse it yourself, without needing any assignments (except in the preamble to define \foo and a helper macro):

\documentclass{book}

%\usepackage{xstring}
%\newcommand{\foo}[1]{\StrBefore{#1}{.}}

\makeatletter
\def\fooex#1.#2\@nil{#1}
\newcommand{\foo}[1]{\fooex #1.\@nil}
\makeatother

\newcounter{try}
\newcommand\test[1]{\setcounter{try}{\foo{#1}}%
\arabic{try}}

\begin{document}

Result of foo\{3.2\}: \foo{3.2},
Result of foo\{32\}: \foo{32},

Result of test\{3.2\}: \test{3.2}

\end{document}

Updated using suggestions from Jonathan: pass an extra . to the helper macro so that something sensible happens when the input string doesn't contain one; use \@nil instead of \relax as the 'special token', now needing \makeatletter protection, as LaTeX itself uses \@nil for similar purpose and it is less likely to appear in the argument (and if it does, is more likely to generate a meaningful error).

This is the stringstrings approach. Things to know about it:

[q] is quiet mode, [v] is verbose mode. The package stores results of string manipulations in the expanded result \thestring, and results of string "queries" in \theresult. So in the macro \test, you can do whatever you want with the expanded \thestring (in this case, I just print it).

\documentclass{book}
\usepackage{stringstrings}

\newcounter{tempdot}

\newcommand{\foo}[2][v]{%
  \whereischar[q]{#2}{.}%
  \setcounter{tempdot}{\theresult}%
  \addtocounter{tempdot}{-1}%
  \substring[#1]{#2}{1}{\thetempdot}%
}

\newcommand\test[1]{%
  \foo[q]{#1}%
  \thestring%
}

\begin{document}
\noindent
Result of foo\{3.2\}: \foo{3.2}\\
Result of test\{3.2\}: \test{3.2}

\end{document}