Problems with mouseout event

I'd give the container div:

position: relative;

and add a third div in the container (should be the last child of the container) with:

position: absolute;
top: 0;
bottom: 0;
left: 0;
right: 0;

and catch the mouseover and mouseout events on this div instead.

Because it has no child elements, you shouldn't get spurious mouseover and mouseout events propagating to it.

Edit:

What I believe happens, is that when the cursor moves from a parent element onto a child element, a mouseout event occurs on the parent element, and a mouseover event occurs on the child element. However, if the mouseover handler on the child element does not catch the event and stop it propagating, the parent element will also receive the mouseover event.

It sounds like what you really want is mouseenter/mouseleave (IE proprietary events, but easy to emulate):

// Observe mouseEnterLeave on mouseover/mouseout
var mouseEnterLeave = function(e) {
    var rel = e.relatedTarget, cur = e.currentTarget;
    if (rel && rel.nodeType == 3) {
        rel = rel.parentNode;
    }
    if(
        // Outside window
        rel == undefined ||
        // Firefox/other XUL app chrome
        (rel.tagName && rel.tagName.match(/^xul\:/i)) ||
        // Some external element
        (rel && rel != cur && rel.descendantOf && !rel.descendantOf(cur))
    ) {
        e.currentTarget.fire('mouse:' + this, e);
        return;
    }
};
$(yourDiv).observe('mouseover', mouseEnterLeave.bind('enter'));
$(yourDiv).observe('mouseout', mouseEnterLeave.bind('leave'));

// Use mouse:enter and mouse:leave for your events
$(yourDiv).observe(!!Prototype.Browser.IE ? 'mouseenter' : 'mouse:enter', yourObserver);
$(yourDiv).observe(!!Prototype.Browser.IE ? 'mouseleave' : 'mouse:leave', yourObserver);

Alternatively, patch prototype.js and use mouseenter and mouseleave with confidence. Note that I've expanded the check for leaving the window or entering XUL chrome; this seemed to fix some edge cases in Firefox for me.