Restrict Directory Size
Solution 1:
I'm not sure of a way to limit the size of a single directory. You could create a new user, assign a quota to them, and then run the process under that user, but I'm guessing that's not what you're after.
As you hint at, you can create a filesystem as a "file" and mount it as the output directory for this app. This would ensure it never spills over to your regular filesystem:
dd if=/dev/zero of=~/disk_image_file count=$size_in_blocks
mkfs -t ext3 -q ~/disk_image_file
mkdir -p ~/mnt/app1/log
mount -o loop=/dev/loop0 ~/disk_image_file ~/mnt/app1/log
Solution 2:
You're looking for quotas. Yes, it is entirely possible to implement quotas on most if not all unix filesystems.
Here's what you should read: http://www.faqs.org/docs/Linux-mini/Quota.html You shouldn't need to do any kernel config. Any vaguely modern system will likely have this enabled already by the distribution.
Come back if you have problems.