how to use grep to match with either whitespace or newline

If you are looking for word AAA followed by space anywhere in the string, or at the end of line, then use

grep -P "AAA( |$)"

Use "AAA\b" if it's acceptable to also match AAA followed by any other non-alphanumeric character. According to the grep man pages, \b matches the empty string at the edge of a word.

$ echo -e "AAA VVV \nVVV AAA\nBBB" | grep "AAA\b"
AAA VVV
VVV AAA

You can use the -e option of grep to select many patterns:

grep -e "AAA$" -e "AAA[[:space:]]"

From the grep man:

-e PATTERN, --regexp=PATTERN
      Use  PATTERN  as  the  pattern.   This  can  be  used to specify
      multiple search patterns, or to protect a pattern beginning with
      a hyphen (-).  (-e is specified by POSIX.)