Why don't we consider the "most general" spin 1 Lagrangian, but only a special case?

The equations of motion (and thus the free particle states) will not be changed if we add a term to the Lagrangian which is in the form of a total divergence of a vector $\partial_\mu W^\mu$. Consider the $C_2$ term and rewrite it, up to divergences, as: $$\partial^\mu A^\nu \partial_\nu A_\mu = \partial^\mu(A^\nu \partial_\nu A_\mu) - A^\nu \partial^\mu \partial_\nu A_\mu = \partial^\mu(A^\nu \partial_\nu A_\mu) - \partial_\nu(A^\nu \partial^\mu A_\mu) - (\partial_\mu A^\mu)^2$$ We can now redefine the quantities in your Lagrangian as $$\epsilon = sign (C_1), \, \eta = sign(C_3), \xi = \frac{|C_1|}{C_1 + C_2}, m^2 = |\frac{C_4}{C_1}|, V_\mu = 2 \sqrt{|C_1|} A_\mu, F_{\mu\nu} = V_{\mu,\nu} - V_{\nu,\mu} $$ to obtain $$\mathcal{L}_\mathrm{Proca} = -\frac{1}{4}\epsilon F^{\mu\nu}F_{\mu\nu} + \frac{m^2}{2} \eta V^\mu V_\mu - \frac{1}{2 \xi} (\partial^\mu V_\mu)^2$$ As an exercise, you can compute the equations of motion of this Lagrangian, decompose the linear equations into a transversal solution $V^\mu_\mathrm{T}$ which has $\partial_\mu V^\mu = 0$, and longitudinal solutions $V^\mu_\mathrm{L}$ which have nonzero divergence, and you will obtain separate equations of motion in the form $$(\Box + m^2) V^\mu_\mathrm{T} = 0$$ $$(\Box + \eta \xi m^2) V^\mu_\mathrm{L} = 0$$ I.e., the transversal solutions are a Proca field of mass $m$. Further analysis shows you that $V^\mu_\mathrm{L}$ is in fact a spin-0 field of mass $\sqrt{\eta \xi} m$, $V^\mu_\mathrm{L} \sim \partial^\mu \phi$. This also explains why we write the Proca kinetic term as $\sim F^{\mu\nu} F_{\mu\nu}$, because it includes only transversal kinetics and $F^{\mu\nu}_\mathrm{L} \sim \partial_\mu \partial_\nu \phi - \partial_\nu \partial_\mu \phi = 0$.

The limit $\xi \to \infty$ makes this scalar mode infinitely heavy and thus inactive, but in some approaches to quantization of Proca field, $\xi$ is left finite, canonical quantization is executed, and only after that you take the $\xi \to \infty$ limit.