So what exactly are perverse sheaves anyway?

If you're looking for a more geometric interpretation of perverse sheaves, you might be interested in MacPherson's 1990 lecture notes "Intersection Homology and Perverse Sheaves." As far as I know, I don't think these were ever officially published, but you can find them on my web site at: http://faculty.tcu.edu/gfriedman/notes/ih.pdf

After some searching I've found the notes An illustrated guide to perverse sheaves, by Geordie Williamson, which is a beautifully illustrated (and sort of topologically oriented) introduction to perverse sheaves.

An $\infty$-categorical perspective is given here https://arxiv.org/abs/1507.03913 and a triangulated expansion of those ideas is here https://arxiv.org/abs/1806.00883

More or less, perverse sheaves are the heart of a certain $t$-structure that you build "gluing along a perversity datum".