Good introduction to statistics from a algebraic point of view?

Lucien Le Cam developed an approach to statistics that largely disposes of measure-theoretic probability and replaced probability measures and random variables with certain Banach lattices. The approach can be found in Le Cam's book Asymptotic Methods in Statistical Decision Theory and the more accessible Comparison of Statistical Experiments by Torgersen.

Keeping the traditional measure theoretic approach to statistics but studying it by a category-theoretic approach is Statistical Decision Rules and Optimal Inference by Cencov.

For basic material on linear regression, there is also The Coordinate-Free Approach to Linear Models by Wichura; this is an area amenable to an approach that is likely to be more comfortable for an algebraist. This is the only book in the list that might be said to be introductory.

That being said, anyone who actually wants to work in statistics needs to be familiar with the standard literature and approach. Warts and all. Much of statistical theory is about inequalities; more analysis than algebra.

P. Diaconis "Group representations in probability and statistics, Chapter 6. Metrics on Groups, and Their Statistical Use" pdf projectEuclid is something eye-opening and must-and-pleasure to read, giving group theoretic look on basic tools in statistics like Mann-Whitney, Kolmogorov-Smirnov tests, Kendall and Spearmen correlations coefficients.

Let me give brief outlook:

The central ideas are related to symmetric group $S_n$, and metrics on it, which give a clue to measuring "disorder" in samples, thus related to main statistics questions.

1) Kendall rank correlation coefficient (tau) is closely related to number of inversions of permutations.

2) Spearman's rank correlation coefficient is closely related to $L_2$-metric on the permutation group.

3) On Mann–Whitney U test and Kolmogorov–Smirnov test let me quote:

Example 13. Rank tests. Doug Critchlow (1986) has recently found a remark able connection between metrics and nonparametric rank tests. It is easy to describe a special case: consider two groups of people — m in the first, n in the second. We measure something from each person which yields a number, say $x_i$, $y_i$ We want to test if the two sets of numbers are "about the same."

This is the classical two-sample problem and uncountably many procedures have been proposed. The following common sense scenario leads to some of the most widely used nonparametric solutions.

Rank all n + m numbers, color the first sample red and the second sample blue, now count how many moves it takes to unscramble the two populations. If it takes very few moves, because things were pretty well sorted, we have grounds for believing the numbers were drawn from different populations. If the numbers were drawn from the same population, they should be well intermingled and require many moves to unscramble.

To actually have a test, we have to say what we mean by "moves" and "unscramble." If moves are taken as "pairwise adjacent transpositions," and unscramble is taken as "bring all the reds to the left," we have a test which is equivalent to the popular Mann-Whitney statistic. If m = n, and moves are taken as the basic insertion deletion operations of Ulam's metric (see Section B below) we get the Kolmogorov-Smirnov statistic

See also Lectures on Algebraic Statistics by Drton, Sturmfels, Bernd, Sullivant: http://www.springer.com/gp/book/9783764389048