Use tee (or equivalent) but limit max file size or rotate to new file

In package apache2-utils is present utility called rotatelogs, it fully meet to your requirements.

Synopsis:

rotatelogs [ -l ] [ -L linkname ] [ -p program ] [ -f ] [ -t ] [ -v ] [ -e ] [ -c ] [ -n number-of-files ] logfile rotationtime|filesize(B|K|M|G) [ offset ]

Example:

your_program | rotatelogs -n 5 /var/log/logfile 1M

Full manual you may read on this link.


or using awk

program | awk 'BEGIN{max=100} {n+=length($0); print $0 > "log."int(n/max)}'

It keeps lines together, so the max is not exact, but this could be nice especially for logging purposes. You can use awk's sprintf to format the file name.

Here's a pipable script, using awk

#!/bin/bash
maxb=$((1024*1024))    # default 1MiB
out="log"              # output file name
width=3                # width: log.001, log.002
while getopts "b:o:w:" opt; do
  case $opt in
    b ) maxb=$OPTARG;;
    o ) out="$OPTARG";;
    w ) width=$OPTARG;;
    * ) echo "Unimplented option."; exit 1
  esac
done
shift $(($OPTIND-1))

IFS='\n'              # keep leading whitespaces
if [ $# -ge 1 ]; then # read from file
  cat $1
else                  # read from pipe
  while read arg; do
    echo $arg
  done
fi | awk -v b=$maxb -v o="$out" -v w=$width '{
    n+=length($0); print $0 > sprintf("%s.%0.*d",o,w,n/b)}'

save this to a file called 'bee', run 'chmod +x bee' and you can use it as

program | bee

or to split an existing file as

bee -b1000 -o proglog -w8 file

use split:

my_program | tee >(split -d -b 100000 -)

Or if you don't want to see the output, you can directly pipe to split:

my_program | split -d -b 100000 -

As for the log rotation, there's no tool in coreutils that does it automatically. You could create a symlink and periodically update it using a bash command:

while ((1)); do ln -fns target_log_name $(ls -t | head -1); sleep 1; done

To limit the size to 100 bytes, you can simply use dd:

my_program | dd bs=1 count=100 > log

When 100 bytes are written, dd will close the pipe and my_program receives EPIPE.