How to log multiple threads in different log files?

For log4j v2 you can use RoutingAppender to dynamically route messages. You can put value for key 'threadId' into the ThreadContext map and then use this id as a part of the file name. There is an example which I have easily applied for the same purpose as yours. See http://logging.apache.org/log4j/2.x/faq.html#separate_log_files

Be aware when putting values into ThreadContext map: "A child thread automatically inherits a copy of the mapped diagnostic context of its parent." So if you have put a value for key 'threadId' into the parent thread and eventually created multiple threads from it, then all child threads will inherit the value of 'threadId' value. I was not able to simply override this value by using put() one more time - you need to use ThreadContext.clear() or explicitly remove() the value from thread context map.

Here is my working log4j.xml:

<?xml version="1.0" encoding="UTF-8"?>
<configuration status="WARN">
    <properties>
        <property name="logMsgPattern">%d{HH:mm:ss} %-5level - %msg%n</property>
        <property name="logDir">test logs</property><!-- ${sys:testLogDir} -->
    </properties>
    <appenders>
         <Console name="Console" target="SYSTEM_OUT">           
            <PatternLayout pattern="${logMsgPattern}"/>
         </Console>

        <Routing name="Routing">
                    <Routes pattern="$${ctx:threadId}">             
                        <Route>
                            <RollingFile name="RollingFile-${ctx:threadId}" fileName="${logDir}/last-${ctx:threadId}.log" filePattern="${logDir}/%d{yyyy-MM-dd}/archived_%d{HH-mm}-${ctx:threadId}.log">
                                    <PatternLayout pattern="${logMsgPattern}"/>
                                    <Policies>
                                <OnStartupTriggeringPolicy />
                            </Policies> 
                    </RollingFile>
                        </Route>
                    </Routes>
            </Routing>  
    </appenders>

    <loggers>               
        <root level="debug">
            <appender-ref ref="Console" level="debug" />
            <appender-ref ref="Routing" level="debug"/>
        </root>                     
    </loggers>  
</configuration>

Logback has a special appender called SiftingAppender which provides a very nice solution to the type of problems you describe. A SiftingAppender can be used to separate (or sift) logging according to any runtime attribute, including thread id.

@havexz 's approach is quite good: writing everything to the same log file and using nested diagnostic contexts.

If your concern is about several JVMs writing to the same FileAppender, then i'd suggest two things:

• using SLF4J as a logging facade
• using logback as logging implementation, in prudent mode

In prudent mode, FileAppender will safely write to the specified file, even in the presence of other FileAppender instances running in different JVMs, potentially running on different hosts.