How can a Software application defend against DoS or DDoS?

Fist of all, it is important to distinguish DoS, which stands for Denial of Service, and DDoS - Distributed Denial of Service. In most cases DoS is caused by software vulnerability, here comes patching and updates as a measure of attack prevention. DoS against service that is not vulnerable are not distributed attacks, they are ineffective and not used today. If to talk about DDoS, which I guess was the point of your question, then things come a bit complicated. IPTABLES in case of huge PPS usually does not help greatly - attack comes from different locations, like in case of zombie-machines. You just ban users, but there will appear new again and again. What I can suggest:

1. configure server software so it handles connections in a more lightweight manner - deny long and heavy requests (e.g. MaxKeepAliveRequests in Apache);
2. configure your OS - IP stack tuning (e.g. sysctl in FreeBSD);
3. close unused ports, remove unused services;
4. mod_evasive - Apache module against attacks;
5. set up a lightweight server as front-end - Nginx, for example.

This is a long topic to discuss, I would recommend not to stop within these recommendations but google on them for more precise information.

Ignoring the part about DDoS for now, since this should mostly be stopped either at the network or by the server infrastructure (I include IIS/Apache/another webserver in that)...

I think it's also important to understand that DoS (at the application layer, and not distributed) can still be further seen as a few different types of attacks, judging by the damage:

• Permanent DoS - Once the attack is mounted, it will not cease until manual (admin) intervention, such as server/app restart, resetting config, deleting data, etc.
• Temporary DoS - Either lasts during the duration of the attack or ends shortly after.
• Flooding - Simply swamping the server with masses of requests, that the server has to handle sequentially, blocking legitimate users from accessing the system. (Most DDoS falls into this category).

Each category has different tactics to attack, and of course, different solutions to prevent. However, there are a few coding flaws that can lead to DoS, but this is FAR from a complete list:

• (Certain types of) Buffer overflows
• SQL Injection - can DROP TABLE
• A loop on user-controlled data - can be made "infinite"
• Not closing DB connections and file handles, and the user can force opening many more of these
• XML Bomb - using DTD entities an attacker can easily use up multiple Gigs of memory and 100% CPU on multi-processor systems (see also http://msdn.microsoft.com/en-us/magazine/ee335713.aspx and https://stackoverflow.com/questions/1906927/xml-vulnerabilities)
• User account lockout - Login mechanisms are recommended to implement a lockout mechanism - i.e., lock the user after 5 wrong passwords. If this is a "permanent" lockout, AND I can discover all usernames (or a significant portion thereof), I can loop through them all and purposefully put the wrong password in 5 times, effectively locking the users out... AND I can do this to the administrator too, so he can't even log in to unlock the other users.
• Small request, requires a very large response, or long processing time, without checking the user is still waiting.
• Single request, kicks off multiple processes, e.g. sending many emails at once.
• Uploading large files to a server with a small disk / small database/mail server/ limited folder quota / etc.
• "Business" DoS - e.g. sending many support requests, that a human will need to sift through

Yes, as you can tell there are many many different patterns and flaws that can lead to DoS or allow flooding. It's not a single class of attack, but rather one category of damage.
Protecting against all of them requires addressing each one and implementing an appropriate control, and then performing Threat Modeling to see if there are other relevant threats that can affect your Availability.

One general recommendation you can consider, depending on your system and NOT a silver bullet for all these attacks (but helpful nonetheless), is implementing a "watchdog" service. This service can monitor the app, restart it if necessary, notify an admin if needed, etc. NOT recommended for just any old' web app though...