How did they find my custom admin URL?

There are a handful of ways your admin url can be exposed. Some include

• Modules that incorrectly create admin controllers. Visiting a known, improper, admin frontname will redirect to the login screen. For instance, hitting example.com/mymodule_admin/foo/bar. A "safe" admin controller will extend on top of your customadmin frontname, like example.com/customadmin/mymodule/foo_bar.
  • There is a fix for this with SUPEE-6788, but it's a setting you have to change manually.
• The URL is visible in the XML response of example.com/index.php/rss/order/NEW/new.
  • Fixed with SUPEE-6285
• Some web hosts create access logs in public areas, like example.com/access_logs. An attacker could look through these logs and sniff out promising URLs.
• Visiting an improperly secured admin controller (e.g. example.com/downloadable/Adminhtml_Downloadable_File/upload)
  • Fixed with SUPEE-5994
• You probably haven't noticed the downloader url (example.com/downloader). Though it's safe behind a login screen, if brute forced an attacker can navigate back to your admin url.

Security through obscurity isn't secure at all. To be safe you should protect your admin interface. This can be done with IP filtering, captchas, rate limits, etc. And, of course, use strong passwords. After all, seeing your admin login screen isn't actually a problem. It's only a problem is an unauthorized user gets in.

The reality is, security by obfuscation nearly never works. I assume it does not even protect you from script kiddies.

But to this case. There are admin modules using their own routes for the admin urls, not using your custom admin url. Payment modules are likely to do this for example(I found 4 of them at our shop).

You can finde some on github with https://github.com/search?p=1&q=AdminController+%22extends+Mage_Adminhtml_Controller_Action%22&ref=searchresults&type=Code&utf8=%E2%9C%93
I assume every controller class not containing _Adminhtml_ is usable for this.

side note, custom url for admin, but not for /downloader ? just brutforce an admin user there, and you get the admin url from there.