In Mario 64 speedruns, why are the keys necessary?

The reason why is because the castle is split into multiple zones (ground floor, basement, and upper castle), which are not loaded at the same time. Like the outside of the castle is a separate zone, and glitching through the door won't actually bring you inside the lobby.

It is possible to get behind the key doors, probably even without TAS, but there is simply nothing there. You have to activate the door to load into the different zone.

The reason why you can skip the first key as you showed, is because the basement has a second entrance, which is not locked. Normally you can't activate a door while underwater, but if you reach the door before the game can register that you are swimming its possible. The upper castle only have one entrance, so this trick can't be used here.

I'm not aware of any glitches where you can load into a different zone through a locked door.

If you are interested in these kinds of technical glitches, you should check out pannenkoek2012 and his second channel on youtube.

There are different categories of speed runs. They are split up based on how many stars are required before finishing the game.

The relevant categories are:

• 0 stars
• 1 star
• 16 stars
• 70 stars
• 120 stars

There are historical reasons behind each of these categories. 120 stars is effectively equivalent to a 100% run in other games, where you get every available star. 70 stars is equivalent to a fast casual run of the game, since 70 stars are the minimum to beat the game without using glitches.

From there, the other 3 categories exist because people found progressively more effective glitches over time for skipping content. First, someone learned how to skip to the top of the infinite stairs and how to glitch through some doors. You needed 15 stars to open one particularly annoying door, then 1 star to move the entrance to Dire Dire Docks back to reveal the Bowser level, then glitch towards the end of the game.

Much later on, someone found a way to skip through the 15-star door, so you only needed to get a single star in Dire Dire Docks to have access to the Bowser Fight. Finally, someone found a way past even that, so you can just fight Bowser without any stars.

Depending on the category of run you watch, different kinds of content will be skipped. However, none of the runs skip any of the Bowser fights (which is how you acquire keys).

After writing all this and re-reading your question, I have realized I don't know why the key doors specifically aren't skipped. I'll leave this up because the background info might shed some light on the puzzle-solving process of getting this game to the state it is, but I don't actually know why the key doors aren't skippable. Maybe someone just hasn't found a skip yet, or maybe you literally can't start a given Bowser fight unless the previous one has been completed because of the way the game is programmed.

This video in particular shows that if you take the obvious strategy of doing a backwards long jump (BLJ) behind the second key door, then you get a black unloaded zone that cannot be crossed:

Here is a print screen screenshot from YouTube screenshot from the video that shows the black zone at the back:

I suspect that the code surrounding this has been thoroughly reverse engineered since it is of such high value, I wonder if someone has uploaded notes somewhere.