How does a crypter for bypassing antivirus detection work?

In some circles, "crypter" and "packer" are synonymous to mean binaries or programs which are self-checking and/or self-modifying. Crypters may, more specifically, mean self-modification that includes encryption and/or code scrambling (see more below).

I suggest that you read Chris Eagle's The IDA Pro Book for an introductory understanding of packers and their disassembly of components and architecture.

Daniel Reynaud's work on packers and self-modifying code is also very fascinating. He makes some distinction between integrity checking, blind writes, code scrambling, and encryption of PE files in this blog post about visualizing packers and self-modifying programs. His crude system of checking for these various technique implementations in packers using dynamic binary instrumentation (not the best way of going about this, but it can be very fast compared to other methods) is available at https://code.google.com/archive/p/tartetatintools/

You could give some of the encoders in the metasploit framework a look. In particular the Polymorphic XOR Additive Feedback Encoder ( Shikata Ga Nai ) might be worth a look, as it is dicussed in the Metasploit Unleashed documentation in regards to bypassing antivirus detection.