How can I provide a great external evaluaton of a faculty member's promotion and tenure case?

For context: I’m in math. Take this advice with a grain of salt if you’re in a discipline with different norms.

A good (meaning useful) letter is typically around 2-3 pages long, although I have seen letters as short as 1 page (not very useful) and as long as 5 pages (useful, but not in proportion to its length and tedious to read).

At least 85-90% of the letter’s length will be devoted to discussing the candidate’s research. It’s appropriate to comment briefly about other aspects of their work such as teaching, service, organizational work, outreach etc, to the extent that you feel such feedback would be helpful, or to the extent you were explicitly asked to discuss those things. Unless you were explicitly asked about them, it’s also appropriate not to discuss anything other than research.

For letters I have written, I would estimate that I spent around 4-8 hours in total doing the review. That time is spread out over several days, which I feel is important to make sure the review is thoughtful and I have time to weigh my words carefully and make sure I am willing to commit to the end result. I am usually familiar with only a part of the candidate’s work, so I spend a good amount of time getting up to speed on their full body of work and trying to estimate its significance. But I think it probably wouldn’t make sense for me to agree to do a review if I felt like I needed more than a day’s work to get to a point where I can meaningfully comment on the significance of their work.

In the letter I usually include both some general comments and at least a few paragraphs on each of 2-3 specific results and papers I am more intimately familiar with. My suggestion is to aim for a good balance between the general and the specific - the people reading the letter need a bit of both.

It’s okay to use some technical language, but don’t overdo it - many of the readers will not be experts in the specific area the candidate is working on. For the benefit of those non-experts, it’s helpful to include language that manages to convey how important and influential the candidate’s work has been without getting into a technical discussion. To some extent you can achieve this by relying on “sociological” data such as how prestigious and well-regarded the journals the candidate publishes in (or conferences they lecture at, etc) are, but try not to put too much emphasis on such things or you will risk coming across as a shallow person who only cares about prestige, and your feedback may be discounted. (Also, as JeffE said in a comment, such sociological indicators of prestige can often be estimated by non-experts directly, so your discussing them is potentially not as helpful as a more expert/technical commentary. But to a limited extent, I think it is appropriate and potentially helpful to comment on prestige.)

Good luck! Writing this sort of letter is a big responsibility, but also a very valuable service. It speaks well of your reputation that you were asked to do it.