how to get good looking copyright and registered symbols

Use the textcomp package, which offers a \textregistered symbol (both serif and sans-serif), different to standard LaTeX which uses \textcircled.

\documentclass{article}
\usepackage{textcomp}
\begin{document}
\textregistered\textcopyright
\sffamily\textregistered\textcopyright
\end{document}

Output:

Here are the original LaTeX definitions from latex.ltx:

\DeclareTextCommandDefault{\textcopyright}{\textcircled{c}}
\DeclareTextCommandDefault{\textregistered}{\textcircled{%
      \check@mathfonts\fontsize\sf@size\z@\math@fontsfalse\selectfont R}}

If designed symbols like those of textcomp wouldn't fit to your text font, you could use \textcircled similarly to create a symbol with the used font together with some correction if necessary, with \raisebox etc.

For ConTeXt, use the \registered{} and \trademark{} macros.

I know it's a very old post but I found another solution which provides an output almost equal to

$^{\tiny{\textregistered}}$

but without using the math environment

Matlab\textsuperscript{\tiny\textregistered}

My minimal working example (MWE) is:

\documentclass{article}
\usepackage[utopia]{mathdesign}

\begin{document}
Before: Matlab\textregistered

After: Matlab\,\textsuperscript{\tiny\textregistered}
\end{document}

My preferred version leaves a bit more non-breaking space between Matlab and the registered symbol.

Many professional fonts have dedicated glyphs for the copyright and registered symbols, so if you are using a font like that you can simply use those glyphs. Some of the fonts available for free with TeX include these symbols - I checked Palantino, Utopia and Charter, they all have them. Most professional fonts I have seen have them.

The \copyright command defined in the TeXBook was a superposition of two characters, which is never going to look as good as a specifically-designed glyph. But copyright symbols are rare in mathematics publishing, apart from the copyright page, so it was probably a case of "good enough", particularly because of the font limitations of early TeX.