Is it possible to become a tenure track Assistant Professor at the same university where I did Post Doc?
I think this might vary by institution, but for most, at least in the US, there wouldn't be any problem with such a move. Of course, you are unlikely to get any advantage in your application and regulations most places will require that you be treated like any other.
I know of one situation like this in which the university really wanted to keep a post-doc so wrote the specification for the tenure track job that matched him nearly perfectly. Nearly everything except the Father's middle name. They sent out the required national search and got back three candidates (among many others) that seemed to be a better fit (on paper) than he was. But he got the job in the end.
So, not impossible (most places) but likely no advantage either.
(I am acquainted with mathematics...) To supplement the other reasonable answers, it may be worth noting that there are generally several times more post-docs available (at R1 universities and high-end R2s) than there are tenure-track jobs at those universities. In my observation the ratio is something like 4 times more post-docs than tenure-track. So, roughly, only a smallish fraction of those post-docs will get tenure-track jobs at comparable institutions.
To get one at the same institution is, in fact, even rarer (statistically), because one tends to move to "somewhat lower status" institutions... in part because the high-end R1 postdocs who do not get high-end R1 tenure-track jobs do often get lower-end R1 TT jobs or high-end R2, squeezing out the people who did post-docs in those places.
Of course, it is not so stupidly simple as this, but statistically it is approximately so, and for obvious reasons. For individuals, of course, things can be wildly different.
So, yes, it is "possible", but statistically the chances are not so good. That is, the majority of post-docs at a given place would not turn out to be competitive for tenure-track jobs there. But, emphatically, this predicts very little for individuals.
If you are a superstar, people will tend to want you regardless of "incest". Too strong/loaded a term in my opinion. It's not like we are talking about that strong of a legal position or social taboo. Just that schools want to get fresh blood and not become 100% bound into one way of thinking. But "freshbloodedness" is a factor, not a go/nogo. [For most people, it is likely a go/nogo in terms of practical statistics. But not in the sense of a rule that is impossible to break.]
On a departmental level, an occasional person having some experience from the school is not the end of the world. Not like their whole program has no fresh blood/ideas. And having a superstar might outweigh the fresh blood concern.
You still have experience from outside the school, based on your grad degree, so it's not like you are 100% "family". But even in that case, there are occasional times when schools decide to let someone go direct from Ph.D. to tenure track at the same school, without even a postdoc (elsewhere, or even at the school). Rare, of course. But if you are a superstar or in demand because of an expanding field with few experts...
A lot will depend on the specifics. For example, does it create an issue of too much weight or internal competition (e.g. two professors doing East Elbonian mosquito protein research)? Or can you do something differentiated enough from your advisor? Or perhaps are they even looking to grow scale in a particular hot topic (e.g. climatology 20 years ago)? Is your advisor about to retire. Etc.
But trumping it all will be how much of a superstar you are. Nothing wrong with asking of course and putting them as one part of your 150 letters (perhaps even having a couple conversations given you are on site). But I would rate likelihood as low unless you are pretty darned special.