Replace comma with newline in sed on MacOS?

Use tr instead:

tr , '\n' < file

Use an ANSI-C quoted string $'string'

You need a backslash-escaped literal newline to get to sed. In bash at least, $'' strings will replace \n with a real newline, but then you have to double the backslash that sed will see to escape the newline, e.g.

echo "a,b" | sed -e $'s/,/\\\n/g'

Note this will not work on all shells, but will work on the most common ones.