How to allow only one product in a cart?

The events catalog_product_type_prepare_full_options and catalog_product_type_prepare_lite_options are your friends

<?xml version="1.0"?>
<config>
    <modules>
        <Fooman_Example>
            <version>0.1.0</version>
        </Fooman_Example>
    </modules>
    <global>
        <models>
            <fooman_example>
                <class>Fooman_Example_Model</class>
            </fooman_example>
        </models>
        <helpers>
            <fooman_example>
                <class>Fooman_Example_Helper</class>
            </fooman_example>
        </helpers>
    </global>
    <frontend>
        <events>
            <catalog_product_type_prepare_full_options>
                <observers>
                    <fooman_example_catalog_product_type_prepare>
                        <class>Fooman_Example_Model_Observer</class>
                        <method>catalogProductTypePrepare</method>
                    </fooman_example_catalog_product_type_prepare>
                </observers>
            </catalog_product_type_prepare_full_options>
        </events>
    </frontend>
</config>

Then in your Observer class

<?php

class Fooman_Example_Model_Observer
{
    public function catalogProductTypePrepare($observer)
    {
        $quote = Mage::getSingleton('checkout/session')->getQuote();
        if($quote->getItemsCount()>=1){
            Mage::throwException('You can only buy one product at a time.');
        }
    }
}

Instead of rewriting a controller (please oh please don't do that), rather, rewrite the addProduct method to account for the limit:

class YourCompany_YourModule_Model_Cart extends Mage_Checkout_Model_Cart
{
    public function addProduct($productInfo, $requestInfo=null){
        if($this->getItemsCount()>1){
            Mage::throwException(Mage::helper('checkout')->__('Cannot add item - cart quantity would exceed checkout the limit of %s per person.', 1));
        }
        parent::addProduct($productInfo, $requestInfo);
    }
}

If you want to get fancy, replace the 1 above with Mage::getStoreConfig('checkout/options/max_cart_qty) and set the following your module's config.xml:

<default>
    <checkout>
        <options>
            <max_cart_qty>1</max_cart_qty>
        </options>
    </checkout>
</default>

That value is now controlled via the XML value. If you want to get really, really fancy, add this to the system.xml of your new module:

<config>
    <sections>
        <checkout>
            <groups>
                <options>
                    <fields>
                        <max_cart_qty translate="label">
                            <label>Maximum Quantity Allowed in Cart (total qty)</label>
                            <frontend_type>text</frontend_type>
                            <sort_order>100</sort_order>
                            <show_in_default>1</show_in_default>
                        </max_cart_qty>
                    </fields>
                </options>
            </groups>
        </checkout>
    </sections>
</config>

Remember that you need to set a <depends>Mage_Checkout</depends> to your module in order to piggyback on its' predefined system configuration.

A possible way is to rewrite the addAction of Mage_Checkout_CartController.

So check if there is already a product in cart and if yes show an appropriate error message. If not you can call the parent method which is doing the complete add-process:

    if (count($this->_getCart()->getProductIds()) > 0) {
        $this->_goBack();
    } else {
        parent::addAction();
    }