Should you use pointers (unsafe code) in C#?

From "The Man" himself:

The use of pointers is rarely required in C#, but there are some situations that require them. As examples, using an unsafe context to allow pointers is warranted by the following cases:

• Dealing with existing structures on disk
• Advanced COM or Platform Invoke scenarios that involve structures with pointers in them
• Performance-critical code

The use of unsafe context in other situations is discouraged.

Specifically, an unsafe context should not be used to attempt to write C code in C#.

Caution:

Code written using an unsafe context cannot be verified to be safe, so it will be executed only when the code is fully trusted. In other words, unsafe code cannot be executed in an untrusted environment. For example, you cannot run unsafe code directly from the Internet.

Reference

If you have to.

Say that you need to false color a large grayscale image, say 2000x2000 pixels. First write the 'safe' version using GetPixel() and SetPixel(). If that works, great, move on. if that proves to be too slow, you may need to get at the actual bits that make up the image (forget about Color matrices for the sake of the example). There is nothing 'bad' about using unsafe code, but it adds complexity to the project and should thus be used only when necessary.