Functional programming and non-functional programming
What is functional programming
There are two different definitions of "functional programming" in common use today:
The older definition (originating from Lisp) is that functional programming is about programming using first-class functions, i.e. where functions are treated like any other value so you can pass functions as arguments to other functions and function can return functions among their return values. This culminates in the use of higher-order functions such as map and reduce (you may have heard of mapReduce as a single operation used heavily by Google and, unsurprisingly, it is a close relative!). The .NET types System.Func and System.Action make higher-order functions available in C#. Although currying is impractical in C#, functions that accept other functions as arguments are common, e.g. the Parallel.For function.
The younger definition (popularized by Haskell) is that functional programming is also about minimizing and controlling side effects including mutation, i.e. writing programs that solve problems by composing expressions. This is more commonly called "purely functional programming". This is made possible by wildly different approaches to data structures called "purely functional data structures". One problem is that translating traditional imperative algorithms to use purely functional data structures typically makes performance 10x worse. Haskell is the only surviving purely functional programming language but the concepts have crept into mainstream programming with libraries like Linq on .NET.
where would I want to use it instead of non-functional programming
Everywhere. Lambdas in C# have now demonstrated major benefits. C++11 has lambdas. There's no excuse not to use higher-order functions now. If you can use a language like F# you'll also benefit from type inference, automatic generalization, currying and partial application (as well as lots of other language features!).
am I correct in thinking that C is a non-functional programming language?
Yes. C is a procedural language. However, you can get some of the benefit of functional programming by using function pointers and void * in C.
One key feature in a functional language is the concept of first-class functions. The idea is that you can pass functions as parameters to other functions and return them as values.
Functional programming involves writing code that does not change state. The primary reason for doing so is so that successive calls to a function will yield the same result. You can write functional code in any language that supports first-class functions, but there are some languages, like Haskell, which do not allow you to change state. In fact, you're not supposed to make any side effects (like printing out text) at all - which sounds like it could be completely useless.
Haskell instead employs a different approach to IO: monads. These are objects that contain the desired IO operation to be executed by your interpreter's toplevel. At any other level they are simply objects in the system.
What advantages does functional programming provide? Functional programming allows coding with fewer potentials for bugs because each component is completely isolated. Also, using recursion and first-class functions allows for simple proofs of correctness which typically mirror the structure of the code.