Can a compass be used in a submarine?
Yes, a compass can be used in a submarine. It is not much different than a ship fir a compass. As the magnetic field is a slowly-varying field, it will not be fully shielded by the hull.
However, there will be magnetic distortion. Magnetic distortion is divided into hard and soft iron terms. The hard iron terms are biases that add to the magnetic field strength and are due to permanent magnets or electrical fields. The hard iron bias is represented as a vector addition to the real (earth's) magnetic field. The soft iron term is a $3\times 3$ matrix that represents the warping of the ambient magnetic field by the presence of ferrous material. It is a gain multiplier and includes cross terms (hence the 3x3 matrix). The equation for the resulting magnetic field is:
$$\mathbf{H}_m = S \mathbf{H}_e + \mathbf{b}$$
where $\mathbf{H}_m$ is the 3D measured magnetic vector, $\mathbf{H}_e$ is the earth's magnetic field, $S$ is the $3\times 3$ soft iron matrix and $\mathbf{b}$ is the hard iron bias vector.
More specifically, we really should be careful about the frames of reference we are dealing with. With $H_m$ in the frame of the submarine, we can write
$$\mathbf{H}_m = S R \mathbf{H}^{(E)}_e +\mathbf{b},$$
where the superscript $(E)$ refers to the frame fixed to the Earth and $R$ is the rotation matrix between the Earth's frame (sometimes called the north-east-down) frame and the frame of the submarine.
Thus, the compass will work, but it will not tell you which way is north without knowing the distortion coefficients. There are methods, and ships use them, to determine the distortion coefficients. It usually requires driving in a circle to map out the distortions.
I believe the answer is "very unlikely." But not for the reason you might think.
The primary reason for the existence of submarines is stealth. Modern submarines are extraordinarily difficult to detect. Very few phenomena penetrate water at any significant depth. Magnetic fields are something that does pass through water well. Since submarines are made out of huge chunks of metal, and have huge electrical systems (every modern submarine has an electric motor for propulsion), and for several other reasons they distort the earth's magnetic field. Thus, the birth of the Magnetic Anomaly Detector (MAD).
A very common countermeasure to MAD adapted by most submarines is to manage their magnetic field, so they "blend in" to their environment. There are a variety of ways to accomplish this; passively (in which case the other answer would apply), to actively (in which case, you will not be able to use a compass.)
For a more detailed discussion of the issues, see the paper: Management of Naval Vessel's Electromagnetic Signatures A Review of Sources and Countermeasures Surface ships have these issues as well, as many sea mines use magnetic anomalies to trigger.