Compiler test cases or how to test a compiler

For the idea to compile a big open source project:

You could take a project that itself has a test suite. Then you compile the project and its test suite and see if the tests pass. To validate these results you compile project and test suite with an other compiler, and run the tests again.

The general practice is to create a large set of small programs that each demonstrate one aspects of the compiler. These will include both program that compile and ones that shouldn't. General the ASM coming out the back end is not checked but rather the program is run and it's output checked. As for how to make sure there are no bugs in the test cases: make them small, as in 5-10 lines each.

These test suites can be very large as in hundreds to thousands of tests (for example: an out of date test suite for the D programming language) and generally include one or more test cases for every bug ever reported.

There are several compiler test suites out there. We've had some luck using the Plum Hall test suite for a C compiler. It consists of a large set of C code specifically written to test against the language standard. It verifies that the compiler can handle the language syntax and semantics.

Good test suites for real languages are expensive to create and maintain. There's a reason that the Plum Hall test suite, which is industry standard for ANSI C, is so bloody expensive.

George Necula's translation validation is a brilliant idea but also quite expensive to implement.

The one thing that's cheap and easy is this: maintain a suite of regression tests, and every time you fix a bug in your compiler, put a suitable test into your regression suites. With compilers, it's unbelievable how easy it is to keep reintroducing the same bug over and over. Disciplined additions to your regression suite will prevent that, and they don't cost much.