Why does parameter expansion with spaces without quotes work inside double brackets "[[" but not inside single brackets "["?

The single bracket [ is actually an alias for the test command, it's not syntax.

One of the downsides (of many) of the single bracket is that if one or more of the operands it is trying to evaluate return an empty string, it will complain that it was expecting two operands (binary). This is why you see people do [ x$foo = x$blah ], the x guarantees that the operand will never evaluate to an empty string.

The double bracket [[ ]], on the other hand, is syntax and is much more capable than [ ]. As you found out, it does not have the "missing operand" issue and it also allows for more C-like syntax with >, <, >=, <=, !=, ==, &&, || operators.

My recommendation is the following: If your interpreter is #!/bin/bash, then always use [[ ]]

It is important to note that [[ ]] is not supported by all POSIX shells, however many shells do support it such as zsh and ksh in addition to bash

The [ command is an ordinary command. Although most shells provide it as a built-in for efficiency, it obeys the shell's normal syntactic rules. [ is exactly equivalent to test, except that [ requires a ] as its last argument and test doesn't.

The double brackets [[ … ]] are special syntax. They were introduced in ksh (several years after [) because [ can be troublesome to use correctly and [[ allows some new nice additions that use shell special characters. For example, you can write

[[ $x = foo && $y = bar ]]

because the entire conditional expression is parsed by the shell, whereas [ $x = foo && $y = bar ] would first be split into two commands [ $x = foo and $y = bar ] separated by the && operator. Similarly double brackets enable things like the pattern matching syntax, e.g. [[ $x == a* ]] to test whether the value of x starts with a; in single brackets this would expand a* to the list of files whose names starts with a in the current directory. Double brackets were first introduced in ksh and are only available in ksh, bash and zsh.

Inside single brackets, you need to use double quotes around variable substitutions, like in most other places, because they're just arguments to a command (which happens to be the [ command). Inside double brackets, you don't need double quotes, because the shell doesn't do word splitting or globbing: it's parsing a conditional expression, not a command.

An exception though is [[ $var1 = "$var2" ]] where you need the quotes if you want to do a byte-to-byte string comparison, otherwise, $var2 would be a pattern to match $var1 against.

One thing you can't do with [[ … ]] is use a variable as an operator. For example, this is perfectly legal (but rarely useful):

if [ -n "$reverse_sort" ]; then op=-gt; else op=-lt; fi
…
if [ "$x" "$op" "$y" ]; then …

In your example

dir="/home/mazimi/VirtualBox VMs"
if [ -d ${dir} ]; then …

the command inside the if is [ with the 4 arguments -d, /home/mazimi/VirtualBox, VMs and ]. The shell parses -d /home/mazimi/VirtualBox and then doesn't know what to do with VMs. You would need to prevent word splitting on ${dir} to get a well-formed command.

Generally speaking, always use double quotes around variable and command substitutions unless you know you want to perform word splitting and globbing on the result. The main places where it's safe not to use the double quotes are:

• in an assignment: foo=$bar (but note that you do need the double quotes in export "foo=$bar" or in array assignments like array=("$a" "$b"));
• in a case statement: case $foo in …;
• inside double brackets except on the right hand side of the = or == operator (unless you do want pattern matching): [[ $x = "$y" ]].

In all of these, it's correct to use double quotes, so you might as well skip the advanced rules and use the quotes all the time.

When should I assign double quotes around variables like "${var}" to prevent problems caused by spaces?

Implicit in this question is

Why isn’t ${variable_name} good enough?

${variable_name} doesn’t mean what you think it does …

… if you think it has anything to do with problems caused by spaces (in variable values). ${variable_name} is good for this:

$ bar=foo
$ bard=Shakespeare
$ echo $bard
Shakespeare
$ echo ${bar}d
food

and nothing else!¹  ${variable_name} doesn’t do any good unless you’re immediately following it with a character that could be part of a variable name: a letter (A-Z or a-z), an underscore (_), or a digit (0-9).  And even then, you can work around it:

$ echo "$bar"d
food

I’m not trying to discourage its use — echo "${bar}d" is probably the best solution here — but to discourage people from relying on braces instead of quotes, or applying braces instinctively and then asking, “Now, do I need quotes, also?”  You should always use quotes unless you have a good reason not to, and you’re sure you know what you’re doing.
_________________
¹   Except, of course, for the fact that the fancier forms of parameter expansion, for example, ${parameter:-[word]} and ${parameter%[word]}, build on the ${parameter} syntax.  Also, you need to use ${10}, ${11}, etc., to reference the 10th, 11th, etc., positional parameters — quotes won’t help you with that.