Using MDC in log4j to dynamically name the log file

This can be accomplished in Logback, the successor to Log4J.

Logback is intended as a successor to the popular log4j project, picking up where log4j leaves off.

See the documentation for Sifting Appender

The SiftingAppender is unique in its capacity to reference and configure nested appenders. In the above example, within the SiftingAppender there will be nested FileAppender instances, each instance identified by the value associated with the "userid" MDC key. Whenever the "userid" MDC key is assigned a new value, a new FileAppender instance will be built from scratch. The SiftingAppender keeps track of the appenders it creates. Appenders unused for 30 minutes will be automatically closed and discarded.

In the example, they generate a separate log file for each user based on an MDC value. Other MDC values could be used depending on your needs.

This is also possible with log4j. You can do this by implementing your own appender. I guess the easiest way is to subclass AppenderSkeleton.

All logging events end up in the append(LoggingEvent event) method you have to implement.

In that method you can access the MDC by event.getMDC("nameOfTheKeyToLookFor");

Then you could use this information to open the file to write to. It may be helpful to have a look at the implementation of the standard appenders like RollingFileAppender to figure out the rest.

I used this approach myself in an application to separate the logs of different threads into different log files and it worked very well.

I struggled for a while to find SiftingAppender-like functionality in log4j (we couldn't switch to logback because of some dependencies), and ended up with a programmatic solution that works pretty well, using an MDC and appending loggers at runtime:

//  this can be any thread-specific string
String processID = request.getProcessID();  

Logger logger = Logger.getRootLogger();

//  append a new file logger if no logger exists for this tag
if(logger.getAppender(processID) == null){

  try{
    String pattern = "%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n";
    String logfile = "log/"+processID+".log";

    FileAppender fileAppender = new FileAppender(
        new PatternLayout(pattern), logfile, true);
    fileAppender.setName(processID);

    // add a filter so we can ignore any logs from other threads
    fileAppender.addFilter(new ProcessIDFilter(processID));

    logger.addAppender(fileAppender);
  }catch(Exception e){
    throw new RuntimeException(e);
  }
}

//  tag all child threads with this process-id so we can separate out log output
MDC.put("process-id", processID);

//whatever you want to do in the thread
LOG.info("This message will only end up in "+processID+".log!");

MDC.remove("process-id");

The filter appended above just checks for a specific process id:

public class RunIdFilter extends Filter {

  private final String runId;

  public RunIdFilter(String runId) {
    this.runId = runId;
  }

  @Override
  public int decide(LoggingEvent event) {
    Object mdc = event.getMDC("run-id");

    if (runId.equals(mdc)) {
      return Filter.ACCEPT;
    }

    return Filter.DENY;
  }
}

Hope this helps a bit.