Retro video games programming
Most of the glitches in that video are tile based glitches, where there are bugs in the collision detection of the tile maps that make up the levels. All levels are made up of square shaped tiles. If you notice mario is always between tiles where he shouldn't be.
Most of the older consoles had some kind of BIOS ROM.
Some of the source code for these are online:
You can read the mostly-commented disassembly to the 7800 BIOS: http://atarihq.com/danb/files/7800bios.asm
The Atari 5200's BIOS source is more interesting, since it does more than just initialize the system and display a splash screen: http://atarihq.com/danb/files/5200BIOS.txt
The Colecovision had an 8K (!) BIOS ROM as well; it's source is here: http://xi6.com/code/coleco/coleco29.asm
The Odyssey II BIOS source is here: http://atarihq.com/danb/files/o2romsrc.txt
The Intellivision had an OS called "exec," can't find a disassembly online, though I did find a bunch of info about it: http://www.intellivisiongames.com/bluesky/hardware/intelli_tech.html#exec and http://www.beeslife.com/faq.htm#_Toc140592020 - it had routines to move sprites, read controllers, and calculate square roots!
Having somewhat worked on an emulator for the NES (I have it decoding some opcodes, but none of the other hardware is emulated), I can maybe share a few answers.
For most games assembler was used. Optimizing compilers, if available for the CPU were nowhere near as good 20-30 years ago as they are today. To get performance, you needed to write in assembler (This even held true on the PC. Parts of Doom are in ASM). All the more so, since the NES CPU ran at less than 2MHz. Also, memory was more expensive then than it is today. The original Mario was stored in about 40k of memory. 16k of that was the actual code, and the remainder was the graphics and sound resources.
Until the 32 bit console era, any sort of operating system, or even built in utilities, on a console was uncommon (Sega CD was the one of the few in the 16 bit era with an actual BIOS, and there was a small program burned into the Game Boy's processor that was responsible for the Nintendo logo scrolling down on power on). See above about size constraints, as a main reason. When inserting the cartridge, the ROM chip in the car was connected directly to the address bus of the CPU. On power on, the CPU would read from a fixed address to get the actual address the program started at, and then jumped to that location and started execution.
As for resources, the NES Dev Wiki has resources concerning the NES hardware, along with programmming references. Zophar's Domain also has technical documents and public domain ROMs for quite a few console (I don't know if I should link to ZD on this site, just google it)