PutFile append file

For those who don't want to override the data in the file but want to append data.

Appending to single file Using ExecuteStreamCommand processor:

It isn't possible with putFile processor but you can use the ExecuteStreamCommand processor to accomplish this.

In command arguments put attributes you want to log speparated by delimiter ${aatr1};${aatr2};${attr3}

In command path put the absolute path of a bash script: /path/logger.sh

logger.sh:

#!/bin/bash
echo "$1|$2|$3">> /path/attributes.log

attibutres.log will append the three attributes line by line. Make sure that the bash script is executable with nifi.

do a chmod 777 logger.sh

Appending to single file Using ExecuteScript Processor:

Try this ECMAscript:

var flowFile = session.get();
var File = Java.type("java.io.RandomAccessFile");
if (flowFile != null) {
    var filename = flowFile.getAttribute("filename");
    /// write to file
    var filePath = "/path/attributes.log" ;
    flowFile = session.putAttribute(flowFile, "filePath", filePath);
     var file = new File(filePath, "rws");
     file.seek(file.length());
        file.write(filename.getBytes());
        file.write("\n".getBytes());
        file.close();
    // Finish by transferring the FlowFile to an output relationship
    session.transfer(flowFile, REL_SUCCESS);
}

Currently there is no way to append data to a file but you can overwrite the file using PutFile.

The PutFile processor writes a file to disk using the the attribute "filename" on the FlowFile. So if you put an UpdateAttribute processor before a PutFile that updates all incoming FlowFiles with to same "filename" then the PutFile processor will write them all with the same file name on disk.

To do this with PutFile, make sure you configure processor property "Conflict Resolution Strategy" to "Replace".

Sorry. New to this. Let me try again...

LogAttribute processor may be an option because it appears that it offers the closest functionality to a file append. But it doesn't appear to be ideal, as it offers few options for directing the output.

There are two other options you can try if you are intent on using the "out of the box" processor functionality rather than developing classes to customize. Use an ExecuteScript processor to execute a Jython, Groovy, or JS, script that modifies the output flowFile to include only the Attributes you need. Follow that with a PutMongo or a PutSQL processor to update a persisted database resource.

Another option if you don't have a database resource at your disposal: use the ExecuteScript processor mentioned above, followed by a PutFile that outputs a uniquely named filename attribute to a directory - say, ${filename}.${uuid}. You'll wind up with a large number of similarly formatted files - one log record per file - which you can then roll up into one file for analysis using Linux line commands or alternatively employ a final ExecuteScript processor in your workflow to roll up every time a file gets processed through your workflow. This last one may not be a good idea, as it is not clear whether it will introduce synchronization and write contention problems if your stream of flowFiles is high.