Is there any sense in which Dirichlet density is "optimal?"

The best way to impress a philosopher is to tell him/her about ultrafilters. A (non-principal) filter on $\mathbb N$ is a set of infinite subsets of $\mathbb N$ closed under intersections and taking super-sets. A maximal filter (under inclusion) is called an ultrafilter. There are plenty of those but nobody saw them since their existence depends on the axiom of choice. For every filter $\omega$ one can define the concept of convergence of sequences of real numbers: a sequence $b_n$ converges to $b$ if for every $\epsilon$ the set of $i$'s such that $|b-b_i|\le \epsilon$ is in $\omega$. If $\omega$ is an ultrafilter, then every bounded sequence of real numbers has unique $\omega$-limit. It is not true if $\omega$ is not an ultrafilter. The smallest interesting filter (called Fréchet filter) consists of all sets with finite complements. The limit corresponding to that filter is the ordinary limit studied in Calculus. You can start from the Fréchet filter and add sets to it to produce bigger and bigger filters. Each filter gives you a Dirichlet-like density. If $\omega$ is an ultrafilter, then all sets will have density (between 0 and 1). Otherwise there will always be sets without density assigned.

Tenenbaum's book on analytic and probabilistic number theory has something to say on this subject. He mentions that it is "easy" to construct a set without Dirichlet density, but unfortunately he doesn't provide such a construction.

A good place to start is with the logarithmic density: Tenenbaum shows that the Dirichlet density exists if and only if

$\lim_{n \rightarrow \infty} \frac{1}{\log n} \sum_{n \in S} \frac{1}{n}$

exists.

The reason naive density fails to exist for the Benford set (call it $B$) is that oscillations between elements in $B$ and elements not in $B$ grow exponentially with $n$. I would conjecture that a set with superexponential gaps does not have a Dirichlet density - for example the set of integers $n$ where the number of digits of $n$ begins with a 1. One might have to sum over a series like $e^{-\log(\log(n))}$ to get convergence.