smarter hyphenation for narrow columns?

I'm not sure why you are using \nohyphens I'd have thought you want more rather than less. Using the linked table as you don't supply code for the example in the question I get this which isn't too bad, although could obviously be tweaked a bit more.

\documentclass[article,a4paper,12pt,twoside]{memoir}

\usepackage{hyphenat}

\setlrmarginsandblock{40mm}{20mm}{*}
\setulmarginsandblock{20mm}{20mm}{*}

\begin{document}

\renewcommand{\arraystretch}{1.5}
\begin{table}
\begin{center}
\raisebox{3in}{\small
\begin{tabular}{|*{3}{>{\hyphenpenalty0 }p{.25\textwidth}|}}
\hline
% \multicolumn{1}{p{.25\textwidth}}{\textbf{Relevance}}
\begin{center}(PM)
\end{center}

Ultimately relevance depends on peer review, and irrelevant content
may be deleted. Mechanisms to ensure that relevant content
\emph{will} be added could be improved. &
\begin{center}(WP)
\end{center}
People contribute articles about what
they're interested in; apart from this, rules like WP:WEIGHT come into
play.&
\begin{center}(DO)
\end{center}

Anyone can upload projects (for ``full projects'', one time approval
is needed), but getting changes into the core requires considerably
more vetting. \\
%\multicolumn{1}{p{.25\textwidth}}{\textbf{Quality}}
Quality control is handled with corrections and the ``orphaning'' mechanism
in case of nonresponsive authors. Some articles are world-writeable,
as in the wiki model. & Automated tools for spam and vandalism detection
combined with a system of editorial oversight, in which Jimmy Wales
has last say. & In addition to bug reports and feature requests handled
through the issue tracker, modules can make use of an automated patch
testing system.\\
%\multicolumn{1}{p{.25\textwidth}}{\textbf{Scalability}}
Peer review is distributed. Links are handled automatically. Caching
is deployed where relevant; in particular, interlinking features are
kept up to date. & The database and other infrastructure is massively
scaled. There are many bots that help with small tasks. & In theory,
anyone can join. Earl Miles, NYCCamp 2012 keynote: ``\emph{There are
no insiders, except Dries; there are no outsiders, only resumes.}''
\\
% \multicolumn{1}{p{.25\textwidth}}{\textbf{Consistency}}
Although automatic links and corrections can help with consistency,
mainly PM relies on standards for proof and expository quality.
&NPOV is the key rule, which works together with templates and other
process tools to maintain community standards about style and
content. & The project issue queues are the place to go when one
module's changes breaks another's. The core of the project has
considerable oversight in these
matters.\\
% \multicolumn{1}{p{.25\textwidth}}{\textbf{Motivation}}
People are solving some of their learning, exposition, and social
needs on the site by writing and reviewing articles and posting in the
forums.& As of 2006, over 50\% of the site had been written by less
than 1\% of the users; these days, paid editing is somewhat
notorious.& Miles continued: ``\emph{To build a resume, find someone
who needs help, and help them. Find something that needs doing, do
it.}'' \\ \hline
\end{tabular}
}
\end{center}
\caption{As typeset ``out of the box''}
\end{table}


\end{document}

If you allow a bit of flexibility on the right margin, and add microtype you get:

\usepackage{microtype}

...

\begin{tabular}{|*{3}{
   >{\hyphenpenalty0 \rightskip0pt plus .3em \emergencystretch2em}%
   p{.25\textwidth}|}}

When trying to typeset text in very narrow measures, it's sometimes best (least worst?!) to use the \sloppy directive, as is done in the following example.

It can also be helpful in such situations to set \righthyphenmin2. (For English-language texts, the default value of this parameter is 3.)

\documentclass[12pt,letterpaper]{article}
\usepackage[showframe,margin=3.38in]{geometry} 
  % \textwidth = 8.5"-2*3.38" = 1.74"
\frenchspacing
\righthyphenmin2
\sloppy
\usepackage{fontspec}
\setmainfont[Ligatures=TeX]{Linux Libertine O}
\begin{document}
\noindent
Quality control is handled with corrections and the ``orphaning'' mechanism in case 
of nonresponsive authors. Some articles are world-writeable, as in the wiki model.
\end{document}