How does a trial version know it has been installed before?

It either stores that information somewhere on your disk (slightly misnamed .DS_Store files - to appear inconspicuous - have been used for this), or could even report some hardware information (e.g. MAC address) to the manufacturer if there's an internet connection during trial installation (or both).

That way, the installer can determine that the same product was already installed either using the information from somewhere on your disk, or by contacting some web server.

It depends on the product in question exactly how it works.

You could try to log all disk access by the installer process using e.g. the opensnoop utility to find out what files are accessed, and try to identify the file containing the trial information.