When should we use asserts in C?

An error code signals runtime behaviour. An assertion is a debugging tool that allows the developer to assert that their assumptions about the program logic are indeed true.

They're two completely different things with different applications.

Error codes are part of your normal program flow. Assertions are only for debugging, and if an assertion is triggered, that means that your program is not written correctly.

assert aborts the process, but is turned into a no-op when the program is compiled with -DNDEBUG, so it's a rather crude debugging tool and nothing more than that. You should only use assert to check for situations that "can't happen", e.g. that violate the invariants or postconditions of an algorithm, but probably not for input validation (certainly not in libraries). When detecting invalid input from clients, be friendly and return an error code.

An example use of assert could be: you've implemented an incredibly smart sorting algorithm and you want to check whether it really sorts. Since the sorting function is supposed to "just work" and therefore doesn't return a value, you can't add error returns without changing the API.

void sort(int *a, size_t n)
{
    recursive_super_duper_sort(a, 0, n);
    assert(is_sorted(a, n));
}

static bool is_sorted(int const *a, size_t n)
{
    for (size_t i=0; i<n-1; i++)
        if (a[i] > a[i+1])
            return false;

    return true;
}

In the long run, you'd really want a proper unit testing framework for this kind of thing instead of assert, but it's useful as a temporary debugging tool.

In general, asserts are for the programmer (i.e. you) to find logic/programming errors before releasing the program to real users. Asserts should not be used for detecting runtime input errors -- use error codes for these.

This is really a matter of taste. Here is my opinion.

The main rule of thumb: an assertion failure is always a bug in the program.

Use an assert to check function parameters if you expect the caller to ensure that the argument is correct and you want to indicate that any other behavior is a bug in the caller. Dividing by zero is, IMO, a very good example.

Use an error code if you expect the caller not to be able to ensure that the argument is correct before calling. For example, it could be very computationally expensive to check the arguments beforehand.

Never use an assert to check user input.