Can I use mem_limit in docker-compose? and How?

Yes. Memory limitation is supported by docker-compose and value can be set as in your example with "m" for megabytes.

It is possible to check what is the memory limit set for running Docker container using "docker stats" command.

If your container name is "repository_1" then use this command:

docker stats repository_1

The result of this will be simillar to this one:

CONTAINER       CPU %    MEM USAGE/LIMIT    MEM %     NET I/O
repository_1    1.93%    4.609 MiB/60 MiB   7.20%     1.832 KiB/648 B

According to documentation, simple

mem_limit: 1000000000

should be enough. I guess, you should drop "m", and use bytes instead of megabytes.