What is the difference between `a[bc]d` (brackets) and `a{b,c}d` (braces)?

The two are quite different.

a[bc]d is a filename pattern (in shells other than fish). It will expand to the two filenames abd and acd if those are names of existing files in the current directory.

• The [...] part is a bracketed expression that matches a single character out of the ones listed (or collating elements when ranges are included). To match the pattern a[bc]d, the character between the strings a and d in a filename must be either a b or a c.

• If abd exists, but acd does not, then it would only expand to abd, and vice versa.

• If neither abd, nor acd exist, depending on the shell and the options, it would trigger an error (original Unix sh, (t)csh, zsh, fish, bash -O failglob) and possibly exit the shell, or leave the pattern unexpanded¹ (Bourne-like and rc-like shells) or expand to nothing (bash/zsh/yash -o nullglob, some older versions of fish, original Unix sh and (t)csh if there are other matching globs in the same command).

a{b,c}d is a brace expansion (in shells that support these). It will expand to the two strings abd and acd.

• The {...} part is a comma-delimited set of strings (in this example; in some shell, it may also be a range such as a..k or 20..25 or more advanced ones like 00..20..2 or 0..20..2%02d), and the expansion is computed by combining each of these strings with the flanking strings a and d. These strings could be longer than a single character and could also be brace expansions themselves.

• The expansion happens regardless of whether these strings corresponds to existing filenames or not.

If you are constructing strings, use a brace expansion. If you are matching filenames, use a filename pattern.

---

^{¹ In this particular case, a[bc]d could happen to be the name of an existing file which is why it's potentially dangerous to use things like rm -f ./*.[ch] in those shells and rm -f ./*.{c,h} is less of a problem.}

a[bc]d is pattern-matching, and is part of the POSIX standard. In POSIX, this is introduced as the "pattern bracket expression". It is documented in section 2.13 of the manual

When unquoted and outside a bracket expression, the following three characters shall have special meaning in the specification of patterns:

?
A question-mark is a pattern that shall match any character.
*
An asterisk is a pattern that shall match multiple characters, as described in Patterns Matching Multiple Characters.
[
The open bracket shall introduce a pattern bracket expression.

Section 2.13.3 also mentions something that it behaves differently from what one would expect for usual regexs when it is used for filename expansion (emphasis by me)

The rules described so far in Patterns Matching a Single Character and Patterns Matching Multiple Characters are qualified by the following rules that apply when pattern matching notation is used for filename expansion:

The slash character in a pathname shall be explicitly matched by using one or more slashes in the pattern; it shall neither be matched by the asterisk or question-mark special characters nor by a bracket expression. Slashes in the pattern shall be identified before bracket expressions; thus, a slash cannot be included in a pattern bracket expression used for filename expansion. If a slash character is found following an unescaped open square bracket character before a corresponding closing square bracket is found, the open bracket shall be treated as an ordinary character. For example, the pattern "a[b/c]d" does not match such pathnames as abd or a/d. It only matches a pathname of literally a[b/c]d.

a{b,c}d is braces expansion, it is not in the specification by POSIX. Here is the corresponding part from the bash manual (emphasis by me):

Brace expansion is a mechanism by which arbitrary strings may be generated. This mechanism is similar to filename expansion (see Filename Expansion), but the filenames generated need not exist. Patterns to be brace expanded take the form of an optional preamble, followed by either a series of comma-separated strings or a sequence expression between a pair of braces, followed by an optional postscript. The preamble is prefixed to each string contained within the braces, and the postscript is then appended to each resulting string, expanding left to right.

According to the comment by @mosvy, this first appeared from csh but the behavior in bash is different from csh and other shells. This type of braces expansion is also present in glob(3).

There is another type of braces expansion {a..z} that only appeared after bash 3.0, and there are more added in bash 4.0.

In a shell where globbing is turned on, execute in a empty folder, the following result is returned

$ echo a[bc]d
a[bc]d
$ echo a{b,c}d
abd acd

In response to @Jesse_b's comment, if you are in an interactive shell and both of them applies, a[bc]d is less trouble typing. For example grep pattern [ab][12].txt.