Seeking algorithm to place maximum number of points within constrained area at minimum spacing?

I do not know any GIS tool to do that, but I have an idea on the algorithm.

First, an approximation of the maximum point number can be obtained with this formula:

Nb = 4.A / Pi.d^2

(where A is the polygon area and d the minimum spacing distance).

Then, to try to locate these points into the polygon, the best pattern is not the square grid but the hexagonal grid. See:

Finally, some optimization techniques using force models could be used to refine the relative positioning of the points.

NB: It is a well known problem in crystallography.

See the thread at https://math.stackexchange.com/questions/15624/distribute-a-fixed-number-of-points-uniformly-inside-a-polygon . In particular, note the reference (in a comment) to "Poisson disk process" and do some Web searching. The connection with the current question is that when you can distribute a given number of points uniformly, then you can systematically increase that number until no more points can be put into the polygon and that solves the problem of maximizing the number of points subject to a minimum distance requirement. (Technically, the two problems are dual optimization problems where the objectives and constraints are interchanged.)

I think this could be thought of as a "packing" problem.

If so, you might want to try a Genetic Algorithm, perhaps one similar to that in On Genetic Algorithms for the Packing of Polygons.