Etiquette of applying to current institution
In general, you can apply if you feel you're qualified. If they don't want to hire you, they'll reject you, but it's not inappropriate to apply.
Tenure-track jobs get huge numbers of applicants, so statistically, it's extremely likely that you will be rejected. If that's going to seriously disturb you and affect your ability to succeed in your current job, then you may want to think twice.
Some departments might prefer "fresh blood", which would put a strike against candidates from inside. On the other hand, if you've been particularly successful and it's clear that you really want to stay and would take the job if offered, that can be a plus, since it reduces uncertainty for the hiring committee.
If you have a trusted contact or mentor within the department, you could talk to them about whether they think you might be a good fit for the job, or vice versa.
If you do apply, then you should make yourself scarce while other candidates are being interviewed. (Unless of course you have already been rejected by then.) Don't go to their talks and don't try to meet them.
There a too many reasons to be rejected to take a refusal personally. You can be rejected even if everyone, in the jury, as a very high esteem of you, just because you research topic will interfere with someone else's, or is not in the lab priority, or they try to hire someone from that university to create a partnership, etc.
So, if you feel qualified and would like the job, I do not see any reason not to apply.
Also, if ever you ask to someone in the jury whether you should apply:
- if he/she says no, then it's maybe not worth applying;
- but he/she will probably say yes, just for you not to take it personally, even though he/she knows they will eventually reject you. So that might be a forced encouragement to apply.
In short: don't take anything personally!