The theory/algorithm knowledge behind the built-in Discretize* and Boundary*

DiscretizeRegion[] and ToElementMesh[] at their core use TriangleLink and TetGenLink to make the mesh. Have a look at the documentation for specifics about these packages. Both ship with source code. As far as literature goes. Have a look at "Delaunay Mesh Generation" by Jonathan Shewchuk (Author) et al.

Package homepage

• Triangle : A Two-Dimensional Quality Mesh Generator and Delaunay Triangulator

• TetGen: A Quality Tetrahedral Mesh Generator and a 3D Delaunay Triangulator

I don't know what algorithms the built-in functions implement, but here are a few books that implement some of the algorithms you seek:

Computational Geometry: Algorithms and Applications

Computational Geometry in C

Discrete and Computational Geometry

Computational Geometry: An Introduction

Finally, there is this one that deals with shape analysis:

Mathematical Tools for Shape Analysis and Description