What is DOCTYPE?
A doctype defines which version of HTML/XHTML your document uses. You would want to use a doctype so that when you run your code through validators, the validators know which version of HTML/XHTML to check against. This page provides a good overview:
Don't forget to add a doctype
Common doctypes you can use are listed here:
Recommended list of DTDs
Which doctype you should go with depends on the code you're using, but to get an idea, try running your code through the W3C validator and use the Document Type drop-down menu in the "More Options" menu to try different doctypes out.
W3C Markup Validation Service
The DOCTYPE tells the consuming user agent (web browsers, web crawlers, validation tools) what type of document the file is. Using it ensures that the consumer correctly parses the HTML as you intended it.
There are several different DOCTYPES for HTML, XHTML, and Framesets and each of these has two modes Strict and Transitional. Strict says that your markup is using the defined standards exactly. See W3C DTDs page for further details.
Quirksmode is basically the layout method from the browser wars days when the standards were much less respected and defined. Generally a standards mode page, that is valid, will layout more consistently across various browsers, but may lack certain features that you require. One such features is the anchor tag's target attribute. The Quirksmode site is a great resource for these differences.
One final thought is that the new HTML5 standard proposes using a very simple DOCTYPE:
<!DOCTYPE html>
Using this DOCTYPE is a forward compatible way to specify that your pages are in standards mode, and are HTML. This is the method that Google uses, and is reasonably easy to remember. I recommend using this DOCTYPE unless you plan to use XHTML.
Basically, the DOCTYPE describes the HTML that will be used in your page.
Browsers also use the DOCTYPE to determine how to render a page. Not including a DOCTYPE or including an incorrect one can trigger quirks mode.
The kicker here is, that quirks mode in Internet Explorer is quite different from quirks mode in Firefox (and other browsers); meaning that you'll have a much harder job, trying to ensure your page renders consistently with all browsers if the quirks mode is triggered, than you will if it is rendered in standards mode.
Wikipedia has a more in-depth summary of the differences in rendering when using various DOCTYPEs. XHTML is enabled by certain DOCTYPEs, and there is quite a bit of debate about the use of XHTML which is covered well in XHTML — myths and reality.
There are subtle differences between different "standards compliant" rendering DOCTYPEs, such as the HTML5 DOCTYPE (<!DOCTYPE html>, prior to HTML5, only known as the "skinny doctype" which does not trigger standardized rendering in older browsers) and other DOCTYPEs such as this one for HTML 4.01 transitional:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">