Chemistry - Why can't a reaction go to completion?
Solution 1:
You alluded to the answer when you mention activation energy. Kinetically the equilibrium constant is $K_e = k_f/k_b$ where $k_f$ and $k_b$ are the forward are reverse reaction rate constants in the reaction $\ce{A <=> B}$. The reason that there is a finite and not zero back reaction rate constant, is that the activation barrier going B to A is not infinitely high. Quite the opposite in fact and while larger than the barrier A to B, (if the reaction is exothermic), may nevertheless be of such a magnitude that the products can return to reactants and thus an equilibrium is established. In this case molecules are continually transforming between one another. The ratio of concentrations at equilibrium is the equilibrium constant.
The reason that the activation barrier can be surmounted is that there is a Boltzmann distribution of energies in all molecules and this means that the number of molecules in either A or B with energy $E_i$ is given by $n_i=n_0\exp(-E_i/(k_BT))$. This is the Boltzmann distribution for molecules with energy $E_i$ and $k_B$ is the Boltzmann constant.
The total number of molecules of type A is given by the sum of those in each level $E_i$ from level 0 to the maximum energy level, which is $$\frac{n_A}{n_0}=\sum_i \exp(-E^A_i/(k_BT))$$
The equation for molecules of type B with energy $E^B_i$ is $$\frac{n_B}{n_0}=\sum_i \exp(-(E^B_i+\Delta E_0)/(k_BT))=\sum_i \exp(-E^B_i/(k_BT))\exp(-\Delta E_0/(k_BT))$$
where $\Delta E_0$ is the difference in energy between the lowest levels of A and B.
At equilibrium the constant K is the ratio of the two concentrations which is therefore the same as the ratio of the two populations and is
$$K=\frac{Z_A}{Z_B}\exp(-\Delta E_0/(k_BT))$$ where Z represents the summations and are called the partition functions. This equation represents the equilibrium constant in terms of the molecules energy levels.
The reason that there is a Boltzmann distribution, but a fixed energy at a give temperature, is that by far the the most likely way of populating a range of energy levels is according to the Boltzmann distribution. In fact the chance of having a Boltzmann distribution is greater than all other possible ways added together.
In classical thermodynamic terms, a plot of free energy G vs extent of reaction $\zeta$ (zeta) which ranges from 0 (entirely A) to 1 (entirely B), has a minimum at some intermediate value of $\zeta$ and this value naturally depends on the particular reaction. The minimum value is where the reaction is at equilibrium and at the minimum the slope of G vs. $\zeta$ is zero which is also when $(\partial G/\partial \zeta)_{T,P} = 0$.
The minimum in G occurs because the entropy term $-TS$ has a minimum as the mole fraction of B increases (at constant T). Initially it is the entropy of A alone in solution, but at equilibrium is a mixture of A and B. If the reaction goes entirely to B then the entropy is that of B alone. The entropy of a mixture is naturally greater than that of one species thus $-TS$ has a minimum vs. $\zeta$.
If $dn_A$ and $dn_B$ of A and B molecules react the the infinitesimal change in zeta is $d\zeta = dn_A-dn_B$. The total change in free energy is $dG=\mu_Adn_A+\mu_Bdn_B$ where $\mu$ is the chemical potential or the free energy / mole. From these two equations the change in free energy of the reaction is $$dG=(\mu_b-\mu_A)d\zeta$$ The reaction proceeds until $(\partial G/\partial \zeta)_{T,P} = 0$ and thus to proceed further expressions for the chemical potential are needed. For a perfect gas $\mu=\mu^0+RT\ln(p) $ thus $$(\partial G/\partial \zeta)_{T,P} = \mu^0_B-\mu^0_A+RT\ln\frac{p_B}{p_A}$$ which can be recast as $$(\partial G/\partial \zeta)_{T,P} = \Delta G^0+RT\ln\frac{p_B}{p_A}$$ and at equilibrium as the derivative is zero $$\Delta G^0=-RT\ln\frac{p_B}{p_A}=-RT\ln K_p$$ which connect the equilibrium constant ($K_P$ for the gas phase reaction), with the free energy.
Edit:
Notes: The shape of the plot of G vs. extent of reaction can be shown by considering a general reaction of the form $\ce{A + B <=> 2C }$ . The Gibbs function is then $$G = n_a\mu_a + n_b\mu_b + n_c\mu_c $$ for $n_i$ moles of species i with chemical potential $\mu_i$. If we assume that the reaction is one of perfect gases then at total pressure P and with mole fraction x, the chemical potential of species i is $$\mu _i = \mu_i^0+RT\ln(P)+RT\ln(x_i) $$ where $\mu_i^0$ is a function of temperature only. G can now be written as $$G= n_a\mu_a^0 + n_b\mu_b^0 + n_c\mu_c^0+RT\ln(P)+RT(n_a\ln(x_a) +n_b\ln(x_b)+n_c\ln(x_c) )$$ If we assume that the total pressure is 1 atm. then the $RT\ln(P)=0$ and can be removed from the equation. Using the relationship between number of moles $n_a=n_b$ and $n_c=2(1-n_a)$ gives $$G= n_a(\mu_a^0 + \mu_b^0-2\mu_c^0) + 2\mu_c^0 + 2RT(n_a\ln(n_a/2) +(1-n_a)\ln(1-n_a) )$$ The $\mu^0$ are the properties of the pure components so do not change during the course of the reaction which means that G is a function of $n_a$ only.
A plot of $G(n_a)-G(0)$ vs. $n_a$ is shown below.
Plotting this way gives a value of zero when $n=0$, which is the condition should all reactants be converted into product, point S. The equilibrium value is at point E. The energy T-R is that due to mixing reactants and R-E that extra energy of mixing when C is formed. The dashed diagonal line is $n_a(\mu_a^0 + \mu_b^0- 2\mu_c^0)$ which is how the reaction would behave if no entropy of mixing is included.
The important point is that there is a minimum in the free energy curve because the creation of a mixture is an irreversible process and this mixing produces a decrease in free energy.
The equilibrium constant is calculated using $-RT\ln(K_p)=2\mu_c^0-\mu_a^0-\mu_b^0$
Solution 2:
Macroscopic view:
$$\Delta G = \Delta G^{\varnothing} + RT\ln Q$$
As the reaction progresses, $Q$ gets larger so $\Delta G$ becomes more positive. At some point, $\Delta G = 0$ but this happens before $Q \rightarrow \infty$.
Alternatively, for a reaction, you want $Q = K_{\mathrm{eq}}$. Since $K_{\mathrm{eq}}$ is finite, you won't react everything.
Microscopic view:
From the viewpoint of microcanonical ensembles, the probability of a single particle being in a state with energy $E$ is proportional to $e^{-\beta E}$, with $\beta$ defined as $\frac{1}{kT}$. This means that the relative probability between two states is $e^{-\beta \Delta E}$ if the two states are separated by energy $\Delta E$. Since the difference is not infinite, there is some nonzero probability of ending up in the less preferred state.
The fundamental issue is that we have lots of relationships to describe reactions, but they happen to deal with ratios and exponentials. In the former, setting the denominator to zero creates an infinity, which we don't have. In the latter, we require the infinity to get to 0, and so we can't.
Solution 3:
In very simplified macroscopic terms, you can think of every reaction having a forwards and backwards direction (being reversible). For some reactions, both directions are almost equally likely (e.g. reaction $(1)$) while for others the forward reaction greatly outweighs the reverse reaction (e.g. reaction $(2)$). But we can and should usually think of all reactions as reversible.
$$\begin{align}\ce{H2 + I2 &<=> 2 HI}\tag{1}\\[0.6em] \ce{CH4 + 2 O2 &-> CO2 + 2 H2O}\tag{2}\end{align}$$
In again simplified terms, the more of one side you have, the more likely the reaction in the other direction will be. Thus, if you have lots of the products, the reverse reaction will be more likely giving you back some reactants. Likewise, if you have lots of reactants, the forward direction will give you a bit of product.
Thermodynamically, this is expressed with the equilibrium constant as shown in equation $(4)$ for reaction $(3)$.
$$\begin{align}\ce{a A + b B &<=> c C + d D}\tag{3}\\[0.6em] K_1 &= \frac{[\ce{C}]^c [\ce{D}]^d}{[\ce{A}]^a [\ce{B}]^b}\tag{4}\end{align}$$
While an equilibrium constant of $0$ seems possible (no products) an equilibrium constant of infinity is not. A principle called microscopic reversibility (on a microscopic level, the forwards route is an identical but mirrored copy of the reverse route; meaning there is no reason to prefer one over the other) tells us that we should be considering any reaction as going into both directions; the equilibrium $K_2$ for the reverse reaction is the inverse of the equilibrium constant of the forwards reaction (eqation $(5)$). Thus, it is impossible for either equilibrium constant to be $0$.
$$K_2 = K_1^{-1}\tag{5}$$
All the explanations above are extremely simplified but in general they hold true.