Polynomials such that they are integer valued except for some integer k?

If a polynomial of degree $n$ takes integer values at $n+1$ consecutive integer argument, then it takes integer values at all integer arguments. This comes from the formula $$\sum_{k=0}^{n+1}(-1)^k \binom{n+1}k f(x+k)=0\tag1$$ valid for polynomials of degree $\le n$. (And $(1)$ comes from the "calculus of finite differences".)

ADDED IN EDIT

For amusement, here is an alternative "mathematics made difficult" argument for the original question. Let $f$ be a polynomial with rational coefficients taking integer values on $\Bbb Z-\{k\}$. Let $p$ be prime. Then $f$ is continuous map from $\Bbb Z_p$ (the $p$-adic integers) to $\Bbb Q_p$ (the $p$-adic numbers). Then $f(\Bbb Z-\{k\})\subseteq \Bbb Z_p$. The closure of $\Bbb Z-\{k\}$ in $\Bbb Q_p$ is $\Bbb Z_p$ and $\Bbb Z_p$ is closed in $\Bbb Q_p$. By continuity, $f(\Bbb Z_p)\subseteq\Bbb Z_p$. In particular $f(k)\in\Bbb Z_p$. As $f(k)\in\Bbb Q$ and $f(k)\in \Bbb Z_p$ for all $p$ then $f(k)\in\Bbb Z$.