Why do we need the new keyword and why is the default behavior to hide and not override?

Good questions. Let me re-state them.

Why is it legal to hide a method with another method at all?

Let me answer that question with an example. You have an interface from CLR v1:

interface IEnumerable
{
    IEnumerator GetEnumerator();
}

Super. Now in CLR v2 you have generics and you think "man, if only we'd had generics in v1 I would have made this a generic interface. But I didn't. I should make something compatible with it now that is generic so that I get the benefits of generics without losing backwards compatibility with code that expects IEnumerable."

interface IEnumerable<T> : IEnumerable
{
    IEnumerator<T> .... uh oh

What are you going to call the GetEnumerator method of IEnumerable<T>? Remember, you want it to hide GetEnumerator on the non-generic base interface. You never want that thing to be called unless you're explicitly in a backwards-compat situation.

That alone justifies method hiding. For more thoughts on justifications of method hiding see my article on the subject.

Why does hiding without "new" cause a warning?

Because we want to bring it to your attention that you are hiding something and might be doing it accidentally. Remember, you might be hiding something accidentally because of an edit to the base class done by someone else, rather than by you editing your derived class.

Why is hiding without "new" a warning rather than an error?

Same reason. You might be hiding something accidentally because you've just picked up a new version of a base class. This happens all the time. FooCorp makes a base class B. BarCorp makes a derived class D with a method Bar, because their customers like that method. FooCorp sees that and says hey, that's a good idea, we can put that functionality on the base class. They do so and ship a new version of Foo.DLL, and when BarCorp picks up the new version, it would be nice if they were told that their method now hides the base class method.

We want that situation to be a warning and not an error because making it an error means that this is another form of the brittle base class problem. C# has been carefully designed so that when someone makes a change to a base class, the effects on code that uses a derived class are minimized.

Why is hiding and not overriding the default?

Because virtual override is dangerous. Virtual override allows derived classes to change the behaviour of code that was compiled to use base classes. Doing something dangerous like making an override should be something you do consciously and deliberately, not by accident.

If the method in the derived class is preceded with the new keyword, the method is defined as being independent of the method in the base class

However if you don't specify either new or overrides, the resulting output is the same as if you specified new, but you will get a compiler warning (as you may not be aware that you are hiding a method in the base class method, or indeed you may have wanted to override it, and merely forgot to include the keyword).

So it helps you to avoid mistakes and explicitly show what you want to do and it makes more readable code, so one can easily understand your code.

It is worth noting that the only effect of new in this context is to suppress a Warning. There is no change in semantics.

So one answer is: We need new to signal to the compiler that the hiding is intentional and to get rid of the warning.

The follow up question is: If you won't / can't override a method, why would you introduce another method with the same name? Because hiding is in essence a name-conflict. And you would of course avoid it in most cases.

The only good reason I can think of for intentional hiding is when a name is forced upon you by an interface.