What is the difference between single-quoted and double-quoted strings in PHP?

Things get evaluated in double quotes but not in single:

$s = "dollars";
echo 'This costs a lot of $s.'; // This costs a lot of $s.
echo "This costs a lot of $s."; // This costs a lot of dollars.

PHP strings can be specified not just in two ways, but in four ways.

1. Single quoted strings will display things almost completely "as is." Variables and most escape sequences will not be interpreted. The exception is that to display a literal single quote, you can escape it with a back slash \', and to display a back slash, you can escape it with another backslash \\ (So yes, even single quoted strings are parsed).
2. Double quote strings will display a host of escaped characters (including some regexes), and variables in the strings will be evaluated. An important point here is that you can use curly braces to isolate the name of the variable you want evaluated. For example let's say you have the variable $type and you want to echo "The $types are". That will look for the variable $types. To get around this use echo "The {$type}s are" You can put the left brace before or after the dollar sign. Take a look at string parsing to see how to use array variables and such.
3. Heredoc string syntax works like double quoted strings. It starts with <<<. After this operator, an identifier is provided, then a newline. The string itself follows, and then the same identifier again to close the quotation. You don't need to escape quotes in this syntax.
4. Nowdoc (since PHP 5.3.0) string syntax works essentially like single quoted strings. The difference is that not even single quotes or backslashes have to be escaped. A nowdoc is identified with the same <<< sequence used for heredocs, but the identifier which follows is enclosed in single quotes, e.g. <<<'EOT'. No parsing is done in nowdoc.

Notes: Single quotes inside of single quotes and double quotes inside of double quotes must be escaped:

$string = 'He said "What\'s up?"';
$string = "He said \"What's up?\"";

Speed:
I would not put too much weight on single quotes being faster than double quotes. They probably are faster in certain situations. Here's an article explaining one manner in which single and double quotes are essentially equally fast since PHP 4.3 (Useless Optimizations toward the bottom, section C). Also, this benchmarks page has a single vs double quote comparison. Most of the comparisons are the same. There is one comparison where double quotes are slower than single quotes.