Android - how to add a custom lock for the recovery and/or bootloader?
Encrypt your filesystem. You'll need to enter a password or PIN on every boot to decrypt it, so as soon as the device is rebooted none of your personal data will be readable. TWRP even supports encrypted filesystems (it prompts you for your password when it starts as well, I think). If your device is at least 4.2.2 then you can even leave ADB on; they'd need to unlock the screen to add their computer to the whitelist, so they can't push apps or read data there either. You'll be able to have root, a custom bootloader, ADB enabled, and still have a relatively secure device.
Even if it would be doable, consider a few things making it extremely unlikely:
- in that early stage, the touchscreen is not yet initialized – so how would you input your PIN/password/pattern to unlock the bootloader before firing up the custom recovery?
- another risk introduced this way: if you break your display, you are effectively knocked-out: even if the problem described above could be worked around, that work-around would be rendered useless at this place.
True, there are things like touch-based custom recoveries. So technically, that would be the place to make it possible: Once the recovery is initialized, it could ask for a code to unlock before offering its real services. This might even include a "fallback mode" for a broken screen using e.g. shake-patterns (accelerometer-based) or NFC (if applyable). But that's nothing we can solve here at this site: you will have to contact the resp. developers for this (twrp and/or clockworkmod, to name two examples).
I recently wiped my phone to factory after I had rooted the phone and got rid of the bloatware I did not want. (Forgot the passcode) I thought I would have to root my phone again and then get rid of factory installed bloatware....again. I was pleasantly surprised to find both that the root was intact AND the bloatware was still gone. Also, my superuser app was still intact. It should have been gone, but was converted into a system APP and so was not wiped during the reset. That got me thinking along a similar line to your question.
What would happen if you were to imbed the Passcode/Pin data into the system data. You could not gain access to the phone by merely wiping it as the pass/PIN data would still be on the phone after the wipe. Of course that would mean that forgetting your pass/pin would effectively brick your phone.
Of course you could just keep ALL of your data on the SDCard and encrypt it there withan app. Then it wouldn't matter if someone flashed your phone and got access because (s)he could not access your sensitive data without BOTH the encryption app AND the app specific pass/pin.