Advice about job applications in the US (mathematics)
Answers to your questions (the numbering is the order you asked them):
You should definitely email people at the university who might be interested in and familiar with your work. But for tenure track positions it is weird to say you want to work with them. Instead, I would just send a low-key email just letting them know that you applied to their position and would be very interested in coming there. The goal is to be informative but not pushy.
It is true that it is both more expensive and more difficult logistically to interview someone who is coming from overseas, but it certainly does happen. They just have to want you enough. For postdocs, there generally isn't an interview, so it isn't an issue. However, some kinds of postdocs are funded by sources like the NSF that only want to support US citizens or permanent residents, so foreign applicants won't be considered for them.
It's not weird at all. Doing so will not affect your chances at getting a tenure-track position. Everyone knows that the job market is tight.
There has to be someone at the university who advocates for hiring you, which means at least they appreciate the kinds of math you do. But certainly there does not have to be someone who works in precisely the same part of mathematics (and in fact often the "needs of the department" involve hiring in fields that are not well-represented there).
My background: I've served on a large number of search committees in pure mathematics at a couple of different private research universities.
I've participated in many searches over the years, and chaired search committees in each of the last two years. My institution is a smaller research intensive university that specializes only in the STEM disciplines. Our mathematics department specializes in applied areas of mathematics. The situation can be somewhat different at other kinds of institutions, but many aspects of this are related to federal employment law and are fairly standardized.
You can contact the search committee if you want to. However, expressing interest in the position, flattering the committee members, etc. will have very little effect on your chances of getting an interview. The committee will have to justify its selections based only on what is in the official application files. If you have a specific question about the position, then it is appropriate to ask that question. The best way to get your application noticed is to submit a complete and clear application that shows that you are well qualified in terms of whatever it is that the committee is looking for. The hard part can be figuring out what the committee really cares about- advertisements are often worded vaguely.
Hiring for a tenure track position almost always involves an on-campus interview. Since travel from outside of the US is expensive, this can be a factor in deciding who to bring to campus for interviews. Some institutions have special rules about paying for international travel. For example, our institution won't pay for international travel for faculty applicants. If you're planning on being in the US for some other reason (attending a conference), then you should mention this in your application cover letter. At another level, there's often a reluctance to hire applicants who must be sponsored for a visa, although many of the applicants on the market are not permanent residents of the US.
This is very common. There's a box to check on the application form. It's not unusual for a search to fail (the top candidates turn down offers.) In that case, the search committee might be asked to go back into the applications and interview candidates for a visiting assitant professor/postdoc position instead.
Most larger departments want to hire assistant professors to join existing research groups. However, at smaller less research focused institutions there's often a desire to diversify the areas of mathematics in the department. The job ad should make it clear what the department wants.
Some further thoughts, as a supplment to the two excellent answers to your bulleted questions. (I've served on multiple hiring committees for a math department in a mid-tier American research university.)
In the US, it is quite typical to submit a job application without: contacting any faculty members there; writing more than a perfunctory cover letter; trying to ascertain how likely you are to be hired by a particular university. This is especially true if you are just finishing a postdoc, as there is no need to explain why you are applying for a different job than the one you have. That said, in your case I'd recommend addressing your specific desire to move to the US in your cover letter.
Indeed, this is what most of the candidates we interviewed did.
By all means, e-mail people or go into more details in your cover letter if you have anything particular to say. It might help you if you can make the case that you are an especially good fit for a particular university. But if you can't think of anything natural to say, then it is okay not to say anything.
You ask how not to be overlooked. One way in which you might help your cause is to try to attend (and, if possible, speak at) as many US conferences as you can. The Joint Mathematics Meetings in January is very popular, especially among job seekers. More specialized conferences are also a great way to get the attention of people who might have the ability to hire you.