Which metals can cause magnetic interference (passively)?

Well, I think that whatever distortion of the magnetic field is caused by paramagnetic materials - such as aluminum - is also caused, with the opposite sign, by diamagnetic materials - such as bismuth (whose effect should be almost exactly opposite to that of aluminum).

I am convinced you may neglect all those materials. Only paramagnetic materials and various steels and similar metals etc. are relevant. See the relative permeability in this table

http://en.wikipedia.org/wiki/Magnetic_permeability#Values_for_some_common_materials

and just ignore all the materials whose $\mu/\mu_0$ is very close to one, on either side. The only materials that have the opposite but comparably important effects as ferromagnetic materials are superconductors whose $\mu$ vanishes. ;-)

Ferromagnets may sustain their field even if the external source disappears: this is probably the biggest source of interference. However, as you point out, even materials without any permanent field could potentially influence the field that you measure by the mobile device.

The main materials that interfere are ferromagnetic and ferrimagnetic, especially those based on iron, nickel, cobalt and manganese.