What to do when you can’t protect mobile app secret keys?
I think I can help resolve your concerns.
So what are we trying to protect? We're trying to protect ourselves from an attacker breaking into user accounts via our web services. Specifically, how might we protect ourselves from an attacker sufficiently motivated to reverse-engineer our mobile app in the wild?
You can't. It's that simple. Trying to do the impossible is what's causing your concern. Instead, design and implement your server under the assumption that the client may be your app or it may be some unknown malicious client. If you can't secure your app with these assumptions, then you will need to buy into shipping an insecure app (likely bad a bad choice) or only have clients in locations where you have physical control (also likely a bad choice).
Let me address some of your specific statements:
I have to assume mobile banking apps have this figured out. But they are not so likely to share that knowledge here!
No. This is wrong. I'm not sure who you think these secretive mobile banking app developers are but I can assure you that many on this site have developed banking apps, mobile or otherwise, and helped secure them.
Our attacker can run a password list against us by spoofing our app's User Login web sequence. Since our attacker is able to observe HTTPS traffic as it exits the app, our attacker knows how to construct the web service request and authentication.
Absolutely right. The attacker can execute a brute-force password attack on your server. Techniques such as account lockout, exponential delay, and CAPTCHA will help defend against this.
We can encrypt the login (and other) information before it leaves the app. This should prevent both sniffing and spoofing, assuming our attacker does not possess the secret keys used in the encryption.
Given that inbound (to the server) traffic is more important to protect, Public Key encryption would be a possibility. Only the server could decrypt it.
Trust SSL. It is well tested and works well. There's no need to have app-based encryption to protect against a MiTM attack. I know it is freaky to hook up a debugging proxy like Fiddler and see your supposedly encrypted traffic in clear text, but it's not really a security problem. It's the user's data, let them see it if they want. And being the server was written with the concept that it can never know what app the client is running, your protocol should be able to withstand examination by a would-be attacker. Note that you must still do proper SSL validation to prevent a 3rd-party MiTM attack.
So skip the HMAC and client-side app encryption when using SSL. Instead, write a robust and secure API. Not because I say so, but because there is no alternative. Your own analysis proved this.
EDIT:
Prevent our attacker from using our Web Services API to run a password list and potentially log in to a member account. We have other brute-force detection measures in place, but we'd like to directly secure our API from such attack.
I previously mentioned account lockouts though the OWASP guide seems less favorable on them due to the fact that they can be used in a DOS. The did mention delaying the response on repeated failed attempts.
The guide goes on to say:
Although brute-force attacks are difficult to stop completely, they are easy to detect because each failed login attempt records an HTTP 401 status code in your Web server logs. It is important to monitor your log files for brute-force attacks-in particular, the intermingled 200 status codes that mean the attacker found a valid password.
While nobody wants their users to be hacked, recognizing and responding to a successful password attack will dramatically reduce the cost of the attack. Monitoring can also allow you to (temporarily) lockout IP addresses to terminate malicious users (though a strong attack will likely come from many IP addresses making this hard to do).
Two-factor authentication reduces the impact of a stolen password.
Prevent our attacker from using our Web Services API to probe our server for other attack possibilities.
Sorry but you can't really do this without physical security. You can embed secrets in your app, obfuscate it, etc... but none of that really makes the app more secure, it just raises the cost of an attack as those can be backward engineered. So whether you implement those strategies or not, you must still design your API as if an attacker has access to your secrets and code.
I'm a big fan of both SAST and DAST. Run static analyses and resolve all high/critical issues before every deployment. The same for a dynamic scan. Vulnerability scanners can also check your site for known vulnerabilities and bad configuration.
Your problem still doesn't make a whole lot of sense.
It sounds like you're trying to protect user accounts from enumeration and brute forcing passwords. This has nothing to do with your client.
To protect against brute-force password attacks, see https://stackoverflow.com/questions/30369529/block-request-for-multiple-unsuccessful-logins-for-a-period-of-time/30382110#30382110
To protect against username enumeration, Return the same "success" message whether the username exists or not for registration or password resets.
Please clearly indicate the actual problem you want to solve, without offering any potential solutions, and we can help you out better.
Your attacker is able to sniff the app traffic because he must be using some proxy tool such as fiddler. That's not an issue with your application. This is how these proxy tools work. As an App developer you should know that and also you should implement security controls accordingly. If you want to avoid this, you might want to use certificate pinning but that is also not foolproof. You should always assume a worst case scenario and develop your code. Put all security controls and business logic at Server Side and do not store any sensitive information at client side.