How to set NLog max file size?

archiveAboveSize should do the trick. It sets the size (bytes) that will be used as a condition for archiving...
If you want to set archiveFileName to timestamp, ${ticks} can be used. But I would prefer combining date with sequence number, for better readability.

You can set archiveNumbering="DateAndSequence" and archiveAboveSize="5000000"

<targets>
  <target xsi:type="File"
          archiveNumbering="DateAndSequence"
          archiveAboveSize="5000000"
          // other config
</targets>

See this note from here if you are using archiveAboveSize

archiveAboveSize - Size in bytes above which log files will be automatically archived. Long Caution: Enabling this option can considerably slow down your file logging in multi-process scenarios. If only one process is going to be writing to the file, consider setting ConcurrentWrites to false for maximum performance. Warning: combining this mode with Archive Numbering Date is not supported. Archive files are not merged. DateAndSequence will work