How do you evaluate reliability in software?

Reliability and robustness are two different attributes of a sytem:

Reliability

The IEEE defines it as ". . . the ability of a system or component to perform its required functions under stated conditions for a specified period of time."

Robustness

is robust if it continues to operate despite abnormalities in input, calculations, etc.

So a reliable system performs its functions as it was designed to within constraints; A robust system continues to operate if the unexpected/unanticipated occurs.

If you have access to any history of the software you're evaluating, some idea of reliability can be inferred from reported defects, number of 'patch' releases over time, even churn in the code base.

Does the product have automated test processes? Test coverage can be another indication of confidence.

Some projects using agile methods may not fit these criteria well - frequent releases and a lot of refactoring are expected

Check with current users of the software/product for real world information.

It depends on what type of software you're evaluating. A website's main (and maybe only) criteria for reliability might be its uptime. NASA will have a whole different definition for reliability of its software. Your definition will probably be somewhere in between.

If you don't have a lot of time to evaluate reliability, it is absolutely critical that you automate your measurement process. You can use continuous integration tools to make sure that you only ever have to manually find a bug once.

I recommend that you or someone in your company read Continuous Integration: Improving Software Quality and Reducing Risk. I think it will help lead you to your own definition of software reliability.