If else statement in a tabularx

The tabularx environment looks for the terminating \end{tabularx} and absorbs everything in between before starting its work.

In your case this makes \else or \fi inside a table cell, which is bad, because a conditional cannot straddle table cells.

It's not clear what would be the reason for having a different number of columns, but just for testing purpose, let's pretend this makes sense.

\documentclass{article}

\usepackage{tabularx}

\newif\iftest
\makeatletter
\newcommand{\checktest}{%
  \iftest\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
}
\makeatother
\testtrue

\begin{document}

\checktest{%
  \begin{tabularx}{\textwidth}{
    X
    >{\centering}m{1.8cm}
    >{\centering}m{1.11cm}
    >{\centering}m{1.1cm}
    >{\centering}m{1.7cm}
    >{\centering}m{1.7cm}
  }
}{%
  \begin{tabularx}{\textwidth}{
    X
    >{\centering}m{1.8cm}
    >{\centering}m{1.11cm}
    >{\centering}m{1.1cm}
    >{\centering}m{1.7cm}
  }
}
A & B & C & D & E \tabularnewline
A & B & C & D & E \tabularnewline
\end{tabularx}

\testfalse

\checktest{%
  \begin{tabularx}{\textwidth}{
    X
    >{\centering}m{1.8cm}
    >{\centering}m{1.11cm}
    >{\centering}m{1.1cm}
    >{\centering}m{1.7cm}
    >{\centering}m{1.7cm}
  }
}{%
  \begin{tabularx}{\textwidth}{
    X
    >{\centering}m{1.8cm}
    >{\centering}m{1.11cm}
    >{\centering}m{1.1cm}
    >{\centering}m{1.7cm}
  }
}
A & B & C & D & E \tabularnewline
A & B & C & D & E \tabularnewline
\end{tabularx}

\end{document}

This way, the conditional is evaluated before tabularx starts working.

The tabularx environment is actually set up to read the body into a macro; see this excerpt from the documentation (section 3 Differences between tabularx and tabular*):

The body of the tabularx environment is in fact the argument to a command ...

The reason for this is because the table is set multiple times in order to get the correct widths of X-columns and capturing the body allows for processing it more than once quite easily. Since the body is captured, you end up with dangling \else or \fis, depending on whether \testtrue or \testfalse.

One way around this is to store the tabularx definition inside a macro and call it outside the \if...\fi construction:

\documentclass{article}

\usepackage{tabularx}
\newif\iftest

\begin{document}

\begin{table}
  \caption{foobar}

  \iftest
    \def\begintabularx{\begin{tabularx}{\linewidth}{ X X }}
  \else
    \def\begintabularx{\begin{tabularx}{\linewidth}{ X }}
  \fi
  \begintabularx
    % Your table here
  \end{tabularx}
\end{table}

\end{document}

The problem is that the tabularx environment which was \begined inside the \if also needs to be \ended inside the \if. (Think of \begin{...}...\end{...} and \if...\fi statements as pairs of brackets: they need to match up in the right order "([])" rather than "([)]", if you see what I mean...).

To meet your additional requirement that there is, really, only one table (not two), in your case I note that the last two columns are the same width. So, one solution is to define \myval to have value 2 if test is true, and 1 if not. You can then use *{\myval}{>{\centering}m{1.7cm}} in the specification of the tabularx which produces 2 or 1 copies of the final column.

If you want columns of different widths at the end, you'd probably need another solution.

\documentclass{article}

\usepackage{tabularx}
\newif\iftest
\testtrue

\newcommand\myval{\iftest2\else1\fi}
\begin{document}


\begin{table}[t!]
    \caption{foobar}
    \label{tab: foo}
        \begin{tabularx}{\textwidth}{X  >{\centering}m{1.8cm} >{\centering}m{1.11cm} >{\centering}m{1.1cm} *{\myval}{>{\centering}m{1.7cm}}}  
        1&2&3&4&5&6      
        \end{tabularx}

\end{table}

\end{document}