Chemistry - Comparing bond angles in carbonyl dichloride and carbonyl dibromide

The reasoning you utilized for comparing the bond angles of $\ce{NH3}$, $\ce{NF3}$, and $\ce{NCl3}$, while sensible, is simplistic and works best only for a few cases where weighing the factors of steric repulsion against bond pair-bond pair repulsions is feasible. It seems that you have encountered this very problem in the case of comparing the bond angles of $\ce{COCl2}$ and $\ce{COBr2}$.

To circumvent these issues, a better method(less prone to failures) would be using Bent's rule. Now, there has been a lot of discussion on this website over what this rule actually is (see here) and why it is physically useful (see this). The Wikipedia article on the same has a wonderful intuitive discussion under the "Justification" tab for why this rule exists. However, I will not be delving into the origins of this rule, and just state outright that the basic premise of this rule is that not all hybrid orbitals are equivalent under a given hybridization scheme. The percentage s-character and p-character (that is, the amount of s-orbital or p-orbital present in a given hybrid orbital as a percentage of the whole) varies as per the electronegativities of the substituents.

A statement of this rule was provided by Henry A. Bent as follows:

Atomic s character concentrates in orbitals directed toward electropositive substituents.

The idea is that the amount of p character will increase (alternatively, the amount of s character will decrease) in those bonds which have more electronegative substituents (that is, the atom other than carbon in the bond), while simultaneously the amount of p character will decrease(alternatively, the amount of s character will increase) in bonds towards less electronegative substituents in such a manner that the net sum of percentage s and p characters over all the bonds in the molecule comes out to be $100$%. This is essentially a redistribution of the orbital energies across bonds to minimize the energy of the system. Further details on this rule can be found in the links mentioned above.

Once you are aware of this rule, applying it to your case becomes a fairly simple task. I will try to compare the $\ce{Cl-C-Cl}$ and $\ce{Br-C-Br}$ angles here, the orders for the $\ce{O-C-Cl}$ and $\ce{O-C-Br}$ angles can be easily inferred from there. From a classical hybridization perspective, the carbon in both these molecules uses $\ce{sp^2}$ hybrid orbitals for bonding, and all the three hybrid orbitals thus generated should possess equivalent atomic orbital's character ($33.33$% s character and $66.66$% p character approximately). But as per Bent's rule, since $\ce{Cl}$ and $\ce{Br}$ are both more electronegative than carbon, they will increase p character towards their direction (that is, in the $\ce{C-Cl}$ and $\ce{C-Br}$ bonds respectively). Since $\ce{Cl}$ is even more electronegative than $\ce{Br}$, the p character increase in the $\ce{C-Cl}$ bonds will be more than that what will happen in the $\ce{C-Br}$ bonds.

Hence, we can say that more p character will be directed in the two $\ce{C-Cl}$ bonds as compared to the two $\ce{C-Br}$ bonds. This directly bears a relation to the bond angle: more is the p character in the constituent bonds, lesser will be the bond angle(you can see this from a fairly easy observation: $\mathrm{sp}$ hybrid orbitals are generally at an angle of $180°$ with $50$% p character, the $\mathrm{sp^2}$ hybrid orbitals are generally at an angle of $120°$ with $66.66$% p character and the $\mathrm{sp^3}$ hybrid orbitals are generally at an angle of $109.5°$ with $75$% p character).

Hence, Bent's rule predicts that the $\ce{Cl-C-Cl}$ bond angle of $\ce{COCl2}$ should be lesser than the $\ce{Br-C-Br}$ of $\ce{COBr2}$. Experimental data says that the $\ce{Cl-C-Cl}$ bond angle of $\ce{COCl2}$ is $111.83 ± 0.11°$, while the $\ce{Br-C-Br}$ of $\ce{COBr2}$ is $112.3 ± 0.4°$. Hence, the prediction made by Bent's rule is confirmed.

Notice that the difference between the given bond angles is quite small (around $1°$). This is also somewhat expected, as the electronegativity difference between chlorine and bromine is also quite small (approximately $0.2$).

The advantage of employing this method is that you don't have to consider conflicting factors like those you stated in your approach. The rule explains these angle variations based on Coulson's theorem, from which the following relation pops out:

$$\cos \theta = \frac{S}{S-1}=\frac{P-1}{P}$$ for $\theta \in (90^\circ,180^\circ)$

($\theta$ stands for the bond angle while ${P}$ and ${S}$ stand for p character and s character respectively)

As you can clearly see, when the p character goes up, the value of $\cos \theta$ also goes up if you closely observe the simplified form $\cos \theta = 1 - \frac{1}{P}$. This means that the value of $\theta$ will decrease for the given range of $\theta$ (as $\cos \theta$ is a decreasing function for $\theta \in (90^\circ,180^\circ)$)

Hence, you can generally predict things quite accurately with Bent's rule without getting confused with contrasting factors, as they are usually taken care of while determining the atomic orbital characters.