How to set Redis max memory?

Yes - to set the memory limit just uncomment the maxmemory line in the .conf file. The default is 0, which means unlimited (until the operating system runs out of RAM and kills the process - I recommend to always set maxmemory to a sane value).

Updated: as @Eric Uldall mentioned in the comments, a CONFIG SET maxmemory <sane value>, followed by a CONFIG REWRITE should also do the trick. This will modify your redis.conf to preserve changes in case of restart

The documentation in the comments call out bytes but I've used extensions such as mb & gb without any issues.

$ grep ^maxmemory /etc/redis-server.conf
maxmemory 8gb
maxmemory-policy allkeys-lru

And to confirm:

$ redis-cli
...
127.0.0.1:6379> config get maxmemory
1) "maxmemory"
2) "8589934592"