Polynomial $P(x)$ with $P(x)\in\mathbb Q$ iff $x\in \mathbb Q$
(Lemma 1) If a polynomial $f\in \mathbb{R}[x]$ satisfies $a\in \mathbb{Q} \implies f(a)\in \mathbb{Q}$. Then $f(x)\in\mathbb{Q}[x]$.
(Lemma 2) There does not exist $f(x)\in \mathbb{Q}[x]$ with $\deg f \geq 2$, such that for every rational number $r$, all roots of $f(x)-r = 0$ are rational.
Using these lemmas, we can prove your conjecture (that is $P(x) = a +b x$ for rational $a,b$). Let $P(x)\in \mathbb{R}[x]$ such that $x\in \mathbb{Q} \iff P(x)\in \mathbb{Q} $, then Lemma 1 says $P(x)\in \mathbb{Q}[x]$. For any rational number, consider the equation $P(x) = r$, it cannot have any irrational solution. Hence Lemma 2 says $\deg P < 2$.
(Proof of Lemma 1) Assume $f(x) = a_n x^n + a_{n-1} x^{n-1} + \cdots + a_0$, then $$\begin{pmatrix}1 & 1 & \cdots & 1 \\ 2^n & 2^{n-1} & \cdots & 1 \\ \vdots & \vdots & \ddots & \vdots \\ (n+1)^n & (n+1)^{n-1} & \cdots & 1 \end{pmatrix}\begin{pmatrix}a_n \\ a_{n-1} \\ \vdots \\ a_0 \end{pmatrix} = \begin{pmatrix}f(1) \\ f(2) \\ \vdots \\ f(n+1) \end{pmatrix} \in \mathbb{Q}^n $$ Since the matrix is invertible (Vandermonde determinant), the proof is completed.
(Proof of Lemma 2) I shall call a sum of a rational number $r$ of the form $$r = r_1+\cdots+ r_k$$ with $r_i \in \mathbb{Q}$ and absolute values of their denominators $\leq|N|$ a decomposition of $r$ with size $N$. Note that any rational number can admit only finitely many decompositions with a fixed size.
Back to the lemma, via multiplying by an integer, we can assume $f(x) = A x^n + B x^{n-1} + \cdots + C$ has integer coefficients. Then for any integer $m$, $f(x)=m$ has all roots rational. We choose distinct $m_1, m_2, ... \in \mathbb{Z}$ such that $C-m_i$ is prime. Then the roots of $f(x)=m_i$ are of the form $\frac{1}{v}$ or $\frac{C-m_i}{v}$, with $v\mid A$. Consider the sum of these roots, for each $m_i$, we obtain a decomposition of $\frac{-B}{A}$ with size $A$. Since $m_i$ are pairwise distinct, these decompositions must also be pairwise distinct. But this contradicts to the finiteness noted above. So such polynomial cannot exist.
Partial Answer: We can show that $P(x)$ can't be a quadratic with rational coefficients. In fact, $P(x)$ can't be a quadratic at all.
Let $P(x)=ax^2+bx+c$, for $a,b,c\in\mathbb{Q}$. Then $x=\frac{-b+\sqrt{m^2+1}}{2a}$, where $m\in\mathbb{Z}$, will be an irrational $x$ with a rational $P(x)$.
Constant coefficient, and Non-zero Coefficients
Clearly, any constant term, $c$, in the polynomial would need to be rational, since $P(0)=a\cdot0^2+b\cdot0+c=c$. We can set $c=0$ since it wouldn't affect whether a certain $P$ fits the criteria. Similarly, we can generate infinite families of polynomials by choosing different $c\in\mathbb{Q}$.
If $a=0$, then $P(x)=bx$, but we already know that linear equations fit our criteria. If $b=0$, then $P(x)=ax^2$ gives a rational value of $P(\sqrt{2})$ so $b\neq0$. Furthermore, if $a=b=0$, then $P(x)=0$ which is trivially rational for all irrational $x$. Hence, $a,b\neq0$ for all $P$ (not just quadratics).
Construction of $x=\frac{-b+\sqrt{m^2+1}}{2a}$
For any $a,b$, we can construct an irrational $x$ such that $P(x)$ is rational. The quadratic formula tells us that $x=\frac{-b\pm\sqrt{b^2-4a(-P(x))}}{2a}$. For $x$ to be irrational, $\sqrt{b^2-4a(-P(x))}$, must also be irrational. So we can choose an $m\in\mathbb{Z}$, $m\neq0$, such that $\sqrt{b^2-4a(-P(x))}=\sqrt{m^2+1}$, where the right-hand-side is irrational since $m^2+1$ can't be a square number. Rearranging this, we get $P(x)=\frac{m^2-b^2+1}{4a}$. We can also substitute $m$ into the expression for $x$ as $x=\frac{-b\pm\sqrt{m^2+1}}{2a}$.
Verification that $P(x)\in\mathbb{Q}$
We can verify this: when we compute $P(x)$, we get $$\begin{aligned} P\left(\frac{-b+\sqrt{m^2+1}}{2a}\right) &=a\left(\frac{-b+\sqrt{m^2+1}}{2a}\right)^2+b\left(\frac{-b+\sqrt{m^2+1}}{2a}\right) \\&=\frac{1}{4a^2}\left(a(b^2-2b\sqrt{m^2+1}+c^2+1)+2ab(-b+\sqrt{m^2+1})\right) \\&=\frac{1}{4a^2}\left(ab^2-2ab\sqrt{m^2+1}+a(m^2+1)-2ab^2+2ab\sqrt{m^2+1}\right) \\&=\frac{a(m^2+1)-ab^2}{4a^2} \\&=\frac{m^2-b^2+1}{4a}\end{aligned}$$.
Further Results
Combining this answer with @pisco125's Lemma 1, we see that $P(x)$ must have rational coefficients, meaning that it can't be a quadratic at all.