Do I need to remove event listeners before removing elements?

Short answer: yes

Long answer: Most browsers handle this correctly and remove those handlers themselves. There are some older browsers (IE 6 and 7, if i recall correctly) that are messing this up. Yes, there could be memory leaks. You should not have to worry about this, but you need to. Have a look at this document.

Just to update the info here. I've been testing various browsers, specifically for memory leaks for circularly dependent event listeners on iframe onload events.

The code used (jsfiddle interferes with memory testing, so use your own server to test this):

<div>
    <label>
        <input id="eventListenerCheckbox" type="checkbox" /> Clear event listener when removing iframe
    </label>
    <div>
        <button id="startTestButton">Start Test</button>
    </div>
</div>

<div>
    <pre id="console"></pre>
</div>

<script>

    (function() {
        var consoleElement = document.getElementById('console');
        window.log = function(text) {
            consoleElement.innerHTML = consoleElement.innerHTML + '<br>' + text;
        };
    }());

    (function() {
        function attachEvent(element, eventName, callback) {
            if (element.attachEvent)
            {
                element.attachEvent(eventName, callback);
            }
            else
            {
                element[eventName] = callback;
            }
        }

        function detachEvent(element, eventName, callback) {
            if (element.detachEvent)
            {
                element.detachEvent(eventName, callback);
            }
            else
            {
                element[eventName] = null;
            }
        }

        var eventListenerCheckbox = document.getElementById('eventListenerCheckbox');
        var startTestButton = document.getElementById('startTestButton');
        var iframe;
        var generatedOnLoadEvent;

        function createOnLoadFunction(iframe) {
            var obj = {
                increment: 0,
                hugeMemory: new Array(100000).join('0') + (new Date().getTime()),
                circularReference: iframe
            };

            return function() {
                // window.log('iframe onload called');
                obj.increment += 1;
                destroy();
            };
        }

        function create() {
            // window.log('create called');
            iframe = document.createElement('iframe');

            generatedOnLoadEvent = createOnLoadFunction(iframe);
            attachEvent(iframe, 'onload', generatedOnLoadEvent);

            document.body.appendChild(iframe);
        }

        function destroy() {
            // window.log('destroy called');
            if (eventListenerCheckbox.checked)
            {
                detachEvent(iframe, 'onload', generatedOnLoadEvent)
            }

            document.body.removeChild(iframe);
            iframe = null;
            generatedOnLoadEvent = null;
        }

        function startTest() {
            var interval = setInterval(function() {
                create();
            }, 100);

            setTimeout(function() {
                clearInterval(interval);
                window.log('test complete');
            }, 10000);
        }

        attachEvent(startTestButton, 'onclick', startTest);
    }());

</script>

If there is no memory leak, the used memory will increase by around 1000kb or less after the tests are run. However, if there is a memory leak, the memory will increase by about 16,000kb. Removing the event listener first always results in lower memory usage (no leaks).

Results:

• IE6 - memory leak
• IE7 - memory leak
• IE8 - no memory leak
• IE9 - memory leak (???)
• IE10 - memory leak (???)
• IE11 - no memory leak
• Edge (20) - no memory leak
• Chrome (50) - no memory leak
• Firefox (46) - hard to say, doesn't leak badly, so maybe just inefficient garbage collector? Finishes with an extra 4MB for no apparent reason.
• Opera (36) - no memory leak
• Safari (9) - no memory leak

Conclusion: Bleeding edge applications can probably get away with not removing event listeners. But I'd still consider it good practice, in spite of the annoyance.