How to count the number of bytes in a file, grouping the same bytes?

This uses od to show one hex value per line, then sorts and counts:

od -t x1 -w1 -v -An mybinaryfile | sort | uniq -c

(-w1 is an extension, it’s not mandated by POSIX.)

Using Perl to unpack the slurped file to a byte array and then use a hash to count unique bytes:

printf '\xA0\x01\x00\xFF\x77\x01\x77\x01\xA0' | 
  perl -0777 -nE '
    @bytes = unpack("C*",$_) 
    }{ 
    $counts{$_}++ for @bytes; 
    for $k (sort { $a <=> $b } keys %counts) {
      printf "%02X: %d\n", $k, $counts{$k}
    }
 '
00: 1
01: 3
77: 2
A0: 2
FF: 1

If a sufficiently recent version of List::MoreUtils is available, you may be able to simplify the counting by using its frequency function.