Sha256 checksum on terminal displaying strange characters

Did you read the description of the package you installed?

This program will read a passphrase from standard input and print a binary (not printable) hash to standard output.

You tried to print the output to console; what you're seeing is expected behavior.

ubuntu 16.04 default max shared memory per user is 64k. I upped it to 1gb for all users, error goes away.

by adding a line to the end of the /etc/security/limits.conf file will fix this. no need to reboot the system, you'll need to logout though.

$ echo test  | sha256 -s test | base64
mlockall: Cannot allocate memory
Warning: couldn't lock memory, are you root?
NyaDNd1pMQRb3N+SYj/4GaZCRLU9DnRtQ4eXNJ1NpXg=

$ ulimit -l
64
$ sudo bash
# echo "*   -     memlock   1048576" >> /etc/security/limits.conf

log out now.

$ ulimit -l
1048576

$ echo test  | sha256 -s test | base64
NyaDNd1pMQRb3N+SYj/4GaZCRLU9DnRtQ4eXNJ1NpXg=

no more tears!!

I'm guessing this warning is about the corner case of having your secrets paged out of memory, and someone finding it latter in tmp space.