Store JSON directly in bash script with variables?

It can be stored safely; generating it is a different matter, since the contents of $bar may need to be encoded. Let a tool like jq handle creating the JSON.

var=$(jq -n --arg b "$bar" '{
  Comment: "Update DNSName.",
  Changes: [
    {
      Action: "UPSERT",
      ResourceRecordSet: {
        Name: "alex.",
        Type: "A",
        AliasTarget: {
          HostedZoneId: "######",
          DNSName: $b,
          EvaluateTargetHealth: false
        }
      }
    }
  ]
}')

You could use a here-doc:

foo=$(cat <<EOF
{"Comment":"Update DNSName.","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"alex.","Type":"A","AliasTarget":{"HostedZoneId":"######","DNSName":"$bar","EvaluateTargetHealth":false}}}]}
EOF
)

By leaving EOF in the first line unquoted, the contents of the here-doc will be subject to parameter expansion, so your $bar expands to whatever you put in there.

If you can have linebreaks in your JSON, you can make it a little more readable:

foo=$(cat <<EOF
{
  "Comment": "Update DNSName.",
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "alex.",
        "Type": "A",
        "AliasTarget": {
          "HostedZoneId": "######",
          "DNSName": "$bar",
          "EvaluateTargetHealth": false
        }
      }
    }
  ]
}
EOF
)

or even (first indent on each line must be a tab)

foo=$(cat <<-EOF
    {
      "Comment": "Update DNSName.",
      "Changes": [
        {
          "Action": "UPSERT",
          "ResourceRecordSet": {
            "Name": "alex.",
            "Type": "A",
            "AliasTarget": {
              "HostedZoneId": "######",
              "DNSName": "$bar",
              "EvaluateTargetHealth": false
            }
          }
        }
      ]
    }
    EOF
)

and to show how that is stored, including quoting (assuming that bar=baz):

$ declare -p foo
declare -- foo="{
  \"Comment\": \"Update DNSName.\",
  \"Changes\": [
    {
      \"Action\": \"UPSERT\",
      \"ResourceRecordSet\": {
        \"Name\": \"alex.\",
        \"Type\": \"A\",
        \"AliasTarget\": {
          \"HostedZoneId\": \"######\",
          \"DNSName\": \"baz\",
          \"EvaluateTargetHealth\": false
        }
      }
    }
  ]
}"

Because this expands some shell metacharacters, you could run into trouble if your JSON contains something like `, so alternatively, you could assign directly, but be careful about quoting around $bar:

foo='{"Comment":"Update DNSName.","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"alex.","Type":"A","AliasTarget":{"HostedZoneId":"######","DNSName":"'"$bar"'","EvaluateTargetHealth":false}}}]}'

Notice the quoting for $bar: it's

"'"$bar"'"
│││    │││
│││    ││└ literal double quote
│││    │└ opening syntactical single quote
│││    └ closing syntactical double quote
││└ opening syntactical double quote
│└ closing syntactical single quote
└ literal double quote