logrotate: keep N newest files, but don't rename

I'm not sure you can do this by using logrotate. Can you run the following as a daily cron job:

rm $(ls -t | sed -e '1,3d')