Delete custom EAV attributes from the database

See Mage_Eav_Model_Entity_Setup::removeAttribute(). It takes two arguments - the first is the entity code, and the second is the attribute code.

Edit - to run from a non-installation scope:

<?php
include 'app/Mage.php';
Mage::app();
$setup = Mage::getResourceModel('catalog/setup','catalog_setup');
$setup->removeAttribute('catalog_product','attr_code');

The first rule of Magento is: Never edit the database directly.
I admit that I broke this rule on numerous occasions, so...
you can delete the attributes from eav_attribute, the constraints with ON DELETE CASCADE should cleanup the rest of the tables.
But I still think you should take the clean way:

$attributeId = 55;
Mage::getModel('catalog/resource_eav_attribute')->load($attributeId)->delete();

It shouldn't take long and you will feel at peace with yourself because you didn't break the rules.
It's not important what method you choose, but backing up your DB is, in both cases.

DELETE FROM eav_attribute WHERE eav_attribute.attribute_code = "some_attr_code";

is a working solution, I used it many times.

Especially if you remove extension and Magento still want to call unexisting attribute model