Adjusting command behavior based on argument type

pgf provides the functionality to check whether \pgfmathresult is a number or not using \pgfmathfloatparsenumber and checking \F of

\pgfmathfloattomacro{\pgfmathresult}{\F}{\M}{\E}

(which separates \pgfmathresult into "components"):

\documentclass{article}
\usepackage{pgf,collcell}% http://ctan.org/pkg/{pgf,collcell}

\newcommand{\ApplyGradient}[1]{%
  \pgfmathfloatparsenumber{#1}% Parse float
  \pgfmathfloattomacro{\pgfmathresult}{\F}{\M}{\E}% Break result into components
  \ifnum\F=3\relax% Test flag for sign/number type (3 = NaN)
    \textcolor{red}{#1}%
  \else%
    \pgfmathsetmacro{\PercentColor}{100.0*(#1+0.2)/1.3}%
    \textcolor{black!\PercentColor}{#1}%
  \fi}
\newcolumntype{R}{>{\collectcell\ApplyGradient}{r}<{\endcollectcell}}

\begin{document}
\begin{tabular}{lRRr}
  p2p           & 0.08 & 0.74 &  10 \\
  p2p like dest & 0.67 & 0.25 & 185 \\
  scan sql      & 0.01 &  NaN &  23 \\
  skype s.d.    & 0.67 & 0.71 &  80
\end{tabular}
\end{document}

You can compare the argument with NaN:

\documentclass[a4paper,12pt]{report}
\usepackage[english]{babel}

\usepackage{pgf}
\usepackage{collcell}

\newcommand{\ApplyGradient}[1]{%
  \ifnum\pdfstrcmp{#1}{NaN}=0
    #1%
  \else
    \pgfmathsetmacro{\PercentColor}{100.0*(#1+0.2)/1.3}%
    \textcolor{black!\PercentColor}{#1}%
  \fi}
\newcolumntype{R}{>{\collectcell\ApplyGradient}{r}<{\endcollectcell}}

\begin{document}
\begin{table}[htbp]
\begin{tabular}{lRRr}
p2p & 0.08 & 0.74 & 10\\
p2p like dest & 0.67 & 0.25 & 185\\
scan sql & 0.01 & NaN & 23\\
skype s.d. & 0.67 & 0.71 & 80\\
\end{tabular}
\end{table}
\end{document}

You can add any formatting you want in the "true" branch of the conditional, for instance \textcolor{red}{#1} instead of just #1.

If you plan to use the macros also with LuaLaTeX or XeLaTeX, then do

\usepackage{pdftexcmds}
\makeatletter
\newcommand{\ApplyGradient}[1]{%
  \ifnum\pdf@strcmp{#1}{NaN}=\z@
    #1%
  \else
    \pgfmathsetmacro{\PercentColor}{100.0*(#1+0.2)/1.3}%
    \textcolor{black!\PercentColor}{#1}%
  \fi}
\makeatletter

Probably not that practical if this is a single use case but if you have quite a few number of NaN's, pgfplotstable does this check internally and you just provide the data and the relevant styles.

\documentclass{standalone}
\usepackage{pgfplotstable}

\pgfplotsset{compat=1.8,%just to supress warnings
     table/adjust the opacity/.style={
        postproc cell content/.append code={%
             \pgfkeysgetvalue{/pgfplots/table/@preprocessed cell content}\pgfmathresult%
             \ifx\pgfmathresult\empty\relax
             \else%
                 \pgfmathparse{100.0*(\pgfmathresult+0.2)/1.3}
                \begingroup\edef\temp{\endgroup%
                 \noexpand\pgfkeys{/pgfplots/table/@cell content/.add={\noexpand\color{black!\pgfmathresult}}{}}%
                }\temp
             \fi%
        },
    }
}



\begin{document}

\pgfplotstabletypeset[
header=false, %Columns have no header rows
every head row/.style={output empty row},% ALso we don't need a header row
display columns/0/.style={string type,column type=l},% First column is treated as text
display columns/1/.style={column type=r,fixed,adjust the opacity}, 
display columns/2/.style={
    column type=r,
    clear infinite,% Clear the infinities and Nans
    adjust the opacity, % Our style
    empty cells with={NaN},% If empty cell place a NaN
},
display columns/3/.style={column type=r}, 
]{
p2p             0.08 0.74 10
{p2p like dest} 0.67 0.25 185
{scan sql}      0.01 NaN  23
{skype s.d.}    0.67 0.71 80
}
\end{document}