Chemistry - pKa differences and reactivity
Solution 1:
An important case of acidity constants changing with solvent, with consequences in organic chemistry, can be found for compounds with hydroxyls, especially water. It is possible to deduce that hydroxyl compounds are more acidic in protic solvents compared to aprotic solvents, which is attributed to an abnormally high stabilization of the resulting conjugate base via strong $\ce{O^{-}\ ^...\ H-O}$ hydrogen bonds.
The difference is most clear for water. Water molecules solvated by water itself have a $\mathrm{p}K_\mathrm{a}$ of 15.7 in ambient conditions, but if the unusual stabilization from hydrogen bonding were removed, they would have a $\mathrm{p}K_\mathrm{a}$ of around 28. In other words, $\ce{H2O}$ is a weaker acid by some 10 orders of magnitude in non-protic solvents compared to water, and correspondingly, its conjugate base $\ce{OH^{-}}$ is a stronger base by the same factor. This means hydroxide ions can effect quantitative deprotonation of even rather weak acids, including the hydrocarbons cyclopentadiene, indene and fluorene, in solvents such as 1,2-dimethoxyethane.
Unfortunately, compounds containing hydroxide ions have somewhat limited use as superbases in non-protic solvents because the solubility of common hydroxide bases such as $\ce{NaOH}$ or $\ce{KOH}$ is rather small. Other hydroxides such as quaternary ammonium hydroxides have appreciable solubility in non-protic solvents, but they are always found in the form of hydrates. Attempts to prepare anhydrous $\ce{R_4N^+OH^{-}}$ inevitably result in decomposition, because the crystallized water solvates the hydroxide ion and tames its reactivity; once the water is removed, the bare hydroxide ion immediately attacks the quaternary ammonium cation creating an alcohol and a tertiary amine.
As mentioned previously, other compounds containing hydroxyls also show changes in $\mathrm{p}K_\mathrm{a}$ comparing water and non-protic solvents, though the difference tends to be smaller. For ethanol, the difference is at most about 4 units of $\mathrm{p}K$.
Solution 2:
Doing anything blindly is, of course, counter productive. If you were to make the mistake of comparing pka values in different solvents then there would be issues ,however, if you compare pka values while maintaining a constant solvent the assumption is fine for qualitative reasoning. The only counter example I could find is a 1,8-Bis(dimethylamino)naphthalene. It's pka was significantly lower in ACN and DMSO than expected. Again, just make sure you compare things with constant solvents and you're fine.
I think the chemists you speak of are just saying " for a given solvent blah dissociates more so than blah" which lets them see where the reaction lies at equilibrium.
Solution 3:
pkA Values are measured in water, so it's only an approach to think that in other solvents they behave in a similar way. The pKa values are also depending on the temperature.
But a good approach is better than nothing.
If the organic solvent is a protic solvent like alcohol (an alcohol is chemically a bit like water) this approach is quit o.k.
Remember that the pKa tells something about the equilibrium and the energy that is needed to move a proton to another molecule (and so to charge a molecule). If a charged molecule is stabilized as it is in a protic environment the equilibrium allows more protons to move around. In contrast a "naked" small positive charge like a proton without any stabilizing environment need so much energy that it can't exist in a normal laboratory - this is already chemistry of a plasma.