Why file is named catalina.out?

Catalina.out simply contains everything that is written to Tomcat's "System.out" and "System.err".

The name "catalina.out" comes from the catalina.sh script that is conventionally used to launch Tomcat. If you want to change the name ... or do something funky with your out/err streams ... you can hack the script, or replace it with an alternative launch script.

The origin of the name "Catalina" is explained here.

---

I have check logging.properties and found nothing about .out.

The redirection of the standard output / error streams to catalina.out happens in the catalina.sh script.

It seems you need to do it through your own log4j configuration and the swallowOutput property. See wiki FAQ. See also some details in this answer.

log4j is just a logging package so if you're not using it, consult you're logging package's javadoc. If you're not using any logging package - you'd better start using.