Remove Custom Project Template that was Automatically Imported

Templates are also stored in "My Documents\Visual Studio 2010\Templates". Under ProjectTemplates and ItemTemplates and so on depending on the language.

EDIT: i just exported a project template. The zip file was created in 2 locations:

 My Documents\Visual Studio 2010\My Exported Templates\mytemplate.zip 
 My Documents\Visual Studio 2010\Templates\ProjectTemplates\mytemplate.zip

deleting both of these also removed the choice from the new project dialog, so I'm not sure what's happening on your machine :(

Try looking under %userprofile%\Application Data\Microsoft\VisualStudio\10.0\ProjectTemplatesCache. This is where the unzipped cache of the template would usually be stored. If it isn't there, try searching for a folder named ProjectTemplatesCache anywhere in your user data directory.