What's the difference between the Browser Object Model and the Document Object Model?
The Browser Object Model is a larger representation of everything provided by the browser including the current document, location, history, frames, and any other functionality the browser may expose to JavaScript. The Browser Object Model is not standardized and can change based on different browsers.
The Document Object Model is standardized and is specific to current HTML document. It is exposed by the Browser Object Model (i.e., DOM is a subset of BOM).
BOM
This is an informal term as there is no W3C or WHATWG standard that mentions it.
One simple definition would be that the BOM encompasses the entire object structure which is accessible via scripting in the browser, beginning with the window object which "contains everything else", since it's the global object.
The window object contains lots of properties (try console.dir( window );). These properties are specified in numerous web-standards. The "core" specification of the window object is as of now still specified in the HTML standard - see here, but I guess it's only a matter of time until the editors decide to transfer this specification into a separate standard. I'm definitively rooting for a "BOM" or "Browser Environment" standard to make things more logical and appropriate.
DOM
This on the other hand is a formal term. You can find definitions of this term in various standards, for instance the DOM4 standard states:
The DOM is a language- and platform neutral interface that allows programs and scripts to dynamically access and update the content and structure of documents.
Notice how the emphasis is on documents. Unlike the BOM which is basically and umbrella term for all the APIs in the browsers, DOM are only those APIs which deal with documents.
A simple definition would be that the DOM is implemented as the document object (which is the root of the DOM tree btw). Basically, the DOM tree (and everything inside it) can be considered part of the DOM. Analogously, everything beyond the DOM-tree is not part of the DOM.
beyond the DOM-tree == all the properties of window except the document object
"Browser Object Model" (BOM) is a term from the early 2000s that didn't catch on and was replaced[1] with the term "Web APIs"
Web APIs are the JavaScript APIs available to web pages: any objects/interfaces, their properties, methods, and events the browser makes available to the page, except for the objects, like String, that are part of JavaScript language itself.
The DOM (Document Object Model), in context of web development, is a subset of Web APIs concerned with manipulation of the structure and contents of web pages and other "documents".
Historically, the DOM was designed as "a platform- and language-neutral interface" with DOM Level 1 specification describing both the ECMAScript (JavaScript) and Java bindings in appendices. You might still use DOM APIs to work with XML/HTML data from outside the browser (e.g. using Xerces in Java), but the "Living Standard" version of the DOM specification is maintained with the focus on the web use-case, and the most recent W3C implementation report includes mainly (if not only) web browsers.
[1] See Google trends for "Browser object model", and how in a modern book (JavaScript Cookbook: Programming the Web) it's only briefly mentioned as 'BOM - see Web API'.
[answer rewritten in 2019]