Is there file format for checksums?

Write the checksum as lowercase hex digits followed by two spaces and then the filename, with one file per line. This is the format used by md5sum and the various sha*sum CLI tools.

$ sha512sum t.txt
d663b43c84ab4ba20040e568d3cb799512fcc00f1053f96f3079345f680b39429adc92f9c8c45fc9ae6053173ddc5b35ade25ae1d5c91e57b809d0c021d5c891  t.txt

$ sha512sum -c <(sha512sum t.txt)
t.txt: OK

TL;DR

The file format is the output of the command you use to generate the digest.

$ shasum -a 512 [FILE] > SHA512SUM # create a checksum file (SHA512SUM)
$ shasum -c SHA512SUM              # verify [FILE] from checksum file
[FILE]: OK                         # output

See these other linux commands for simple ways to generating a checksum file

Long answer

It depends on which implementation of shamum you are using to check your checksum file.

If you are using a perl implementation ($ shamum -c [FILE]) then you can find the answer in the man pages. see $ man shasum

When checking, the input should be a former output of this program. The default mode is to print a line with checksum, a character indicating type ('*' for binary,' ' for text, '?' for portable, '^' for BITS), and name for each FILE.

If you are using a GNU implementation ($ sha512sum -c [FILE]) then the answer can be found under their coreutils documentation:

NOTE: The GNU sha digests documentation reference the md5sum digest documentation

For each file, ‘md5sum’ outputs by default, the MD5 checksum, a space, a flag indicating binary or text input mode, and the file name. Binary mode is indicated with ‘*’, text mode with ‘ ’ (space). Binary mode is the default on systems where it’s significant, otherwise text mode is the default. If file contains a backslash or newline, the line is started with a backslash, and each problematic character in the file name is escaped with a backslash, making the output unambiguous even in the presence of arbitrary file names. If file is omitted or specified as ‘-’, standard input is read.

All schecksum files I've seen were just plain ASCII files which contained the checksum only. This way they may be easily compared with checksum command outputs.

Good practice is to indicate the checksum algorithm used with a filename suffix, eg .md5 or .crc.