How to find out deadlock and prevent it in C#

It sounds like you had problems explaining how deadlocks can occur and how they can be prevented.

A deadlock occurs when each thread (minimum of two) tries to acquire a lock on a resource already locked by another. Thread 1 locked on Resources 1 tries to acquire a lock on Resource 2. At the same time, Thread 2 has a lock on Resource 2 and it tries to acquire a lock on Resource 1. Two threads never give up their locks, hence the deadlock occurs.

The simplest way to avoid deadlock is to use a timeout value. The Monitor class (system.Threading.Monitor) can set a timeout during acquiring a lock.

Example

try{
    if(Monitor.TryEnter(this, 500))
    {
        // critical section
    }
}
catch (Exception ex)
{

}
finally
{
    Monitor.Exit();
}

Read More

Performance analysis tools can be also helpful in identifying deadlocks, among others. This question will provide some insight in this topic: C#/.NET analysis tool to find race conditions/deadlocks .

Visual analysis of the code and the proper using of locks is useful also (you should be able to detect potential problems in code while inspecting it), but can be very difficult for complex applications. Sometimes the deadlocks are visible only when you run the code, not simply by inspecting the code.

I don't know too much of your interviewer. Some may want to see how much you know of locking standards/guideliness, some may want to see if you know how to use your tools, some may want both. In the company I work at, for example, the use of tools (expecially the ones we already own and use) is highly appreciated. But that does not imply one should not have the skills that would prevent coding deadlocks in the first place.

Locking something just for the sake of locking affects performance, as thread wait for each other. You have to analyse the workflow to determine what really needs to be locked, when, with what type of lock (simple lock or maybe a ReaderWriterLockSlim). There are many typical ways to prevent deadlock.

For example when using ReaderWriterLockSlim you can use a timeout to prevent deadlocks (if you wait to much, you abort aquiring the lock)

if (cacheLock.TryEnterWriteLock(timeout))
{
...
}

And you should be able to suggest such timeouts.

At such a question I would expect at least a mention of the classic case of deadlocks, like badly use of nested locks (you should be able to know can you avoid them, why they are bad, etc.).

The subject is very big... you can go on and on about this. But without definitions. Knowing what a lock is and knowing to use locks/semaphores/mutex in large scale multi-threading applications are 2 different things.

I think the interview is asking you a trick question. If you could use static analysis to prevent deadlock... nobody would have deadlock!

Personally, when I look for deadlock I start by finding functions where the critical section spans more than the function call. For example

void func(){
    lock(_lock){
        func2();
     }
}

It's not really clear what func2 is doing. Maybe it dispatches an event on the same thread, which would mean that event is still part of the critical section. Maybe it then locks on a different lock. Maybe it dispatches into the threadpool and no longer is re-entrant because its now on a different thread! These kinds of places are where you can start to see deadlock scenarios happening: when you have multiple non-reentrant lock locations.

Other times, when tracing deadlock scenarios, I backtrack and I try and find where are all the threads created. I give thought to each function and where it can actually be running. If you aren't sure, adding in logging to log where the function call came from can also help.

You can also avoid deadlocks by using lock free data structures (but those require just as much though to use). You want to minimize your access to the lock free structure because each time you access it it can change.

As mentioned in another answer, you can use mutexes with timeouts, but that isn't guaranteed to always work (what if your code needs to work longer than the timeout?). It was mentioned in another comment that this is maybe what the interviewer was asking for. I find that in production this isn't really a good idea. Timeouts vary all the time, maybe something took longer than expected to run and hit a timeout. I think its better to let it deadlock, take a process dump, then find exactly what was holding the locks and fix the problem. Of course, if your business requirements can't allow that, then you can use this as part of a defensive coding strategy along with smart lock placement choices.

I don't agree with your interview that locks always add a huge performance problem. Uncontended locks/mutexes/etc first test as a spinlock before handing off to the OS and spinlocks are cheap.

In general, the best way to avoid deadlock is to understand your program flow. Each time you introduce a new lock object think about where it is used and what uses it down the chain.